DSL

metametric.dsl

This module contains the domain-specific language (DSL) for defining metrics.

Hook

Bases: ABC, Generic[Tc]

A hook that is called when a match is found.

from_callable(func) staticmethod

Creates a hook from a callback.

on_match(data_id, pred_path, pred, ref_path, ref, score) abstractmethod

Called when a match is found.

Match dataclass

Bases: Generic[T]

Represents a match between a pair of inner objects in a prediction and a reference.

__str__()

Returns a string representation of the match.

Matching

Bases: Iterable[Match[object]]

An object that can be used to iterate over matches and run hooks on them.

__iter__()

Traverses all matching pairs of inner objects.

run_with_hooks(hooks, data_id=0)

Runs hooks on the matches.

Metric

Bases: ParameterizedMetric[T, float], ABC, Generic[T]

The basic metric interface.

Here a metric is defined as a function \(\phi: T \times T \to \mathbb{R}_{\ge 0}\) that takes two objects and returns a non-negative number that quantifies their similarity. It follows the common usage in machine learning and NLP literature, as in the phrase "evaluation metrics". This is not the metric in the mathematical sense, where it is a generalization of distances.

contramap(f_pred, f_ref=None)

Returns a new metric \(\phi^\prime\) by first preprocessing the objects by a given function \(f: S \to T\).

\[ \phi^\prime(x, y) = \phi(f(x), f(y)) \]

Parameters:

Name	Type	Description	Default
f_pred	`Callable[[S], T]`	A function that preprocesses the predicted objects.	required
f_ref	`Callable[[S], T]`	A function that preprocesses the reference objects. If not provided, the preprocessing function for the predicted objects will be used.	None

Returns:

Type	Description
Metric[S]	A new metric \(\phi^\prime\).

from_function(f) staticmethod

Create a metric from a function \(f: T \times T \to \mathbb{R}_{\ge 0}\).

Parameters:

Name	Type	Description	Default
f	`Callable[[T, T], float]`	A function that takes two objects and returns a float. This is the function that derives the metric.	required

Returns:

Type	Description
Metric[T]	Metric[T]: A metric that uses the function to score two objects.

gram_matrix(xs, ys)

Computes the Gram matrix of the metric given two collections of objects.

Parameters:

Name	Type	Description	Default
xs	Sequence[T]	A collection of objects \(\{x_1, \ldots, x_n\}\).	required
ys	Sequence[T]	A collection of objects \(\{y_1, \ldots, y_m\}\).	required

Returns:

Type	Description
Float[ndarray, 'nx ny']	A Gram matrix \(G\) where \(G = \begin{bmatrix} \phi(x_1, y_1) & \cdots & \phi(x_1, y_m) \\ \vdots & \ddots & \vdots \\ \phi(x_n, y_1) & \cdots & \phi(x_n, y_m) \end{bmatrix}\).

family(metric, reduction)

Creates a metric family.

from_func(func)

Create a metric from a binary function.

macro_average(normalizers)

Macro-average reduction.

micro_average(normalizers)

Micro-average reduction.

suite(collection)

Creates a metric suite.