DSL

metametric.dsl

This module contains the domain-specific language (DSL) for defining metrics.

Hook

Bases: ABC, Generic[Tc]

A hook that is called when a match is found.

from_callable(func) staticmethod

Creates a hook from a callback.

on_match(data_id, pred_path, pred, ref_path, ref, score) abstractmethod

Called when a match is found.

Match dataclass

Bases: Generic[T]

Represents a match between a pair of inner objects in a prediction and a reference.

__str__()

Returns a string representation of the match.

Matching

Bases: Iterable[Match[object]]

An object that can be used to iterate over matches and run hooks on them.

__iter__()

Traverses all matching pairs of inner objects.

run_with_hooks(hooks, data_id=0)

Runs hooks on the matches.

Metric

Bases: Generic[T]

The basic metric interface.

Here a metric is defined as a function \(\phi: T \times T \to \mathbb{R}_{\ge 0}\) that takes two objects and returns a non-negative number that quantifies their similarity. It follows the common usage in machine learning and NLP literature, as in the phrase "evaluation metrics". This is not the metric in the mathematical sense, where it is a generalization of distances.

compute(x, y) abstractmethod

Scores two objects using this metric, and returns the score and a matching object.

contramap(f)

Returns a new metric \(\phi^\prime\) by first preprocessing the objects by a given function \(f: S \to T\).

\[ \phi^\prime(x, y) = \phi(f(x), f(y)) \]

Parameters:

Name	Type	Description	Default
f	Callable[[S], T]	A preprocessing function.	required

Returns:

Type	Description
Metric[S]	A new metric \(\phi^\prime\).

from_function(f) staticmethod

Create a metric from a function \(f: T \times T \to \mathbb{R}_{\ge 0}\).

Parameters:

Name	Type	Description	Default
f	`Callable[[T, T], float]`	A function that takes two objects and returns a float. This is the function that derives the metric.	required

Returns:

Type	Description
Metric[T]	Metric[T]: A metric that uses the function to score two objects.

gram_matrix(xs, ys)

Computes the Gram matrix of the metric given two collections of objects.

Parameters:

Name	Type	Description	Default
xs	Sequence[T]	A collection of objects \(\{x_1, \ldots, x_n\}\).	required
ys	Sequence[T]	A collection of objects \(\{y_1, \ldots, y_m\}\).	required

Returns:

Type	Description
ndarray	A Gram matrix \(G\) where \(G = \begin{bmatrix} \phi(x_1, y_1) & \cdots & \phi(x_1, y_m) \\ \vdots & \ddots & \vdots \\ \phi(x_n, y_1) & \cdots & \phi(x_n, y_m) \end{bmatrix}\).

score(x, y)

Scores two objects using this metric: \(\phi(x, y)\).

score_self(x)

Scores an object against itself: \(\phi(x, x)\).

In many cases there is a faster way to compute this than the general pair case. In such cases, please override this function.

family(metric, reduction)

Creates a metric family.

from_func(func)

Create a metric from a binary function.

macro_average(normalizers)

Macro-average reduction.

micro_average(normalizers)

Micro-average reduction.

preprocess(func, m)

Preprocess the input by some function then apply a metric.

This is the contramap operation on the metric functor.

suite(collection)

Creates a metric suite.