mindformers.core.PerplexityMetric

class mindformers.core.PerplexityMetric

Perplexity is defined as the exponentiated average negative log-probability assigned by the model to each word in the test set. Mathematically, for a sequence of words \(W = (w_1, w_2, \ldots, w_N)\) , the perplexity (PP) is given by:

\[PP(W) = P(w_1, w_2, \ldots, w_N)^{-\frac{1}{N}} = \sqrt[N]{\frac{1}{P(w_1, w_2, \ldots, w_N)}}\]

Where \(P(w_1, w_2, \ldots, w_N)\) is the probability of the sequence under the model.

In practical terms, perplexity can be rewritten as:

\[PP(W) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log P(w_i | w_1, w_2, \ldots, w_{i-1})\right)\]

This equation highlights that a lower perplexity indicates a better-performing language model, as it suggests that the model assigns higher probabilities to the actual sequence of words.

Examples

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindformers.core.metric.metric import PerplexityMetric
>>> x = Tensor(np.array([[[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]]))
>>> y = Tensor(np.array([[1, 0, 1]]))
>>> mask = Tensor(np.array([[1, 1, 1]]))
>>> metric = PerplexityMetric()
>>> metric.clear()
>>> metric.update(x, y, mask)
>>> perplexity = metric.eval()
>>> print(perplexity)
'loss': 0.8262470960617065, 'PPL': 2.284728265028813}

clear()

Clearing the internal evaluation result.

eval()

Computing the evaluation result.

Returns

A dict of evaluation results with loss and PPL scores.

update(*inputs)

Updating the internal evaluation result.

Parameters

*inputs (List) – Logits, labels, and input_mask. Logits is a tensor of shape \([N,S,W]\) with data type Float16 or Float32, Labels and input_mask is a tensor of shape \([N,S]\) with data type Int32 or Int64. where \(N\) is the batch size, \(S\) is the sequence length, and \(W\) is the vocabulary size.