Function Differences with tf.keras.metrics.Accuracy, tf.keras.metrics.BinaryAccuracy, tf.keras.metrics.CategoricalAccuracy, and tf.keras.metrics.SparseCategoricalAccuracy
tf.keras.metrics.Accuracy
tf.keras.metrics.Accuracy(
name='accuracy', dtype=None
)
For more information, see tf.keras.metrics.Accuracy.
tf.keras.metrics.BinaryAccuracy
tf.keras.metrics.BinaryAccuracy(
name='binary_accuracy', dtype=None, threshold=0.5
)
For more information, see tf.keras.metrics.BinaryAccuracy.
tf.keras.metrics.CategoricalAccuracy
tf.keras.metrics.CategoricalAccuracy(
name='categorical_accuracy', dtype=None
)
For more information, see tf.keras.metrics.CategoricalAccuracy.
tf.keras.metrics.SparseCategoricalAccuracy
tf.keras.metrics.SparseCategoricalAccuracy(
name='sparse_categorical_accuracy', dtype=None
)
For more information, see tf.keras.metrics.SparseCategoricalAccuracy.
mindspore.train.Accuracy
mindspore.train.Accuracy(eval_type="classification")
For more information, see mindspore.train.Accuracy.
Usage
Accuracy is usually used to calculate accuracy in binary and multiclassification scenarios. MindSpore provides an interface for this function as mindspore.train.Accuracy
, while TensorFlow provides more options with the same overall functionality and computational logic, but the interface has slightly different format requirements for input data.
In the two frameworks, the evaluation function interface computes and updates the accuracy of the current batch sample using the update
method and the update_state
method, which require passing in y_pred and y_true, respectively. Assuming that the current batch data is N and the category is C, the interfaces are roughly distinguished as follows:
MindSpore: mindspore.train.Accuracy
supports both regular single-label scenarios and multi-label scenarios, with the distinction between ‘classification’ and ‘multilabel’ set via the interface parameter eval_type
. In the update
method of mindspore, y_pred defaults to probability values in the range [0,1], with a shape of (N,C); y_true needs to be distinguished according to the scenario: For single label, the shape of y can be (N,C) and (N,), and for multi-label, the shape needs to be (N,C). Please refer to the API notes on the official website for details.
TensorFlow version 1.15 provides numerous interfaces for computing Accuracy, and none of these interfaces in this version support multi-label scenarios. The following interfaces can all be configured with sample weights via sample_weight
, with the following general differences:
tf.keras.metrics.Accuracy
: Both y_true and y_pred are category labels (int) by default, and can be used in multi-category cases.tf.keras.metrics.BinaryAccuracy
: y_true defaults to the category label (int), y_pred defaults to the prediction probability, and the threshold is set by the entry threshold, which is usually used in the binary-category case.tf.keras.metrics.CategoricalAccuracy
: y_true defaults to the onehot encoding of the category, and y_pred defaults to the predicted probability, which is usually used in multi-category cases.tf.keras.metrics.SparseCategoricalAccuracy
: y_true defaults to the category label (int), and y_pred defaults to the predicted probability, which is usually used in multi-category cases.
Code Example
TensorFlow:
tf.keras.metrics.Accuracy:
import tensorflow as tf
import numpy as np
tf.enable_eager_execution()
y_pred = [1, 2, 3, 4]
y_true = [0, 2, 3, 4]
acc = tf.keras.metrics.Accuracy()
acc.update_state(y_true, y_pred)
print(acc.result().numpy())
# out:0.75
tf.keras.metrics.BinaryAccuracy:
y_true = [1, 1, 0, 0]
y_pred = [0.98, 0.7, 0, 0.6]
acc = tf.keras.metrics.BinaryAccuracy()
acc.update_state(y_true, y_pred)
print(acc.result().numpy())
# out: 0.75
tf.keras.metrics.CategoricalAccuracy
y_pred = [[0.2, 0.5, 0.4], [0.3, 0.1, 0.4], [0.9, 0.6, 0.4]]
y_true = [[0, 1, 0], [0, 0, 1], [0, 1, 0]]
acc = tf.keras.metrics.CategoricalAccuracy()
acc.update_state(y_true, y_pred)
print(acc.result().numpy())
# 0.6666667
tf.keras.metrics.SparseCategoricalAccuracy:
y_pred = [[0.2, 0.5, 0.4], [0.3, 0.1, 0.4], [0.9, 0.6, 0.4]]
y_true = [1, 2, 1]
acc = tf.keras.metrics.SparseCategoricalAccuracy()
acc.update_state(y_true, y_pred)
print(acc.result().numpy())
# 0.6666667
MindSpore:
mindspore.train.Accuracy:
import numpy as np
from mindspore.train import Accuracy
import mindspore as ms
# classification
y_pred = ms.Tensor(np.array([[0.2, 0.5, 0.4], [0.3, 0.1, 0.4], [0.9, 0.6, 0.4]]), ms.float32) # 1 2 0
y_true1 = ms.Tensor(np.array([1, 2, 1]), ms.float32) # y_true1: index of category
y_true2 = ms.Tensor(np.array([[0, 1, 0], [0, 0, 1], [0, 1, 0]]), ms.float32) # y_true2: one hot encoding
acc = Accuracy('classification')
acc.update(y_pred, y_true1)
print(acc.eval())
# 0.6666666666666666
acc.clear()
acc.update(y_pred, y_true2)
print(acc.eval())
# 0.6666666666666666
# multilabel:
y_pred = ms.Tensor(np.array([[0, 1, 0], [1, 0, 1], [0, 1, 1]]), ms.float32)
y_true = ms.Tensor(np.array([[0, 1, 1], [1, 0, 1], [0, 1, 0]]), ms.float32)
acc = Accuracy('multilabel')
acc.update(y_pred, y_true)
print(acc.eval())
# 0.3333333333333333