Evaluation Metrics
When a training task is complete, an evaluation function (Metric) is often required to evaluate the quality of a model. Different training tasks usually require different metric functions. For example, for a binary classification problem, common evaluation metrics include precision, recall, and the like. For a multiclass classification task, macro and micro may be used for evaluation.
MindSpore provides evaluation functions for most common tasks, such as nn.Accuracy, nn.Precision, nn.MAE, and nn.MSE. The evaluation functions provided by MindSpore cannot meet the requirements of all tasks. In most cases, you need to customize metrics for a specific task to evaluate the trained model.
The following describes how to customize metrics and how to use metrics in nn.Model.
For details, see Evaluation Metrics.
Customized Metrics
The customized metric function needs to inherit the nn.Metric parent class and re-implement the clear, update, and eval methods in the parent class.
clear: initializes related internal parameters.update: receives network prediction output and labels, computes errors, and updates internal evaluation results after each step.eval: computes the final evaluation result after each epoch ends.
The mean absolute error (MAE) algorithm is shown in formula (1):
The following uses the simple MAE algorithm as an example to describe the clear, update, and eval functions and their usage.
import numpy as np
import mindspore as ms
from mindspore import nn
class MyMAE(nn.Metric):
def __init__(self):
super(MyMAE, self).__init__()
self.clear()
def clear(self):
"""Initialize variables _abs_error_sum and _samples_num."""
self._abs_error_sum = 0 # Save error sum.
self._samples_num = 0 # Accumulated data volume.
def update(self, *inputs):
"""Update _abs_error_sum and _samples_num."""
y_pred = inputs[0].asnumpy()
y = inputs[1].asnumpy()
# Compute the absolute error between the predicted value and the actual value.
abs_error_sum = np.abs(y - y_pred)
self._abs_error_sum += abs_error_sum.sum()
# Total number of samples
self._samples_num += y.shape[0]
def eval(self):
"""Compute the final evaluation result."""
return self._abs_error_sum / self._samples_num
# The network has two outputs.
y_pred = ms.Tensor(np.array([[0.1, 0.2, 0.6, 0.9], [0.1, 0.2, 0.6, 0.9]]), ms.float32)
y = ms.Tensor(np.array([[0.1, 0.25, 0.7, 0.9], [0.1, 0.25, 0.7, 0.9]]), ms.float32)
error = MyMAE()
error.clear()
error.update(y_pred, y)
result = error.eval()
print(result)
0.1499999612569809
Using Metrics in Model Training
mindspore.Model is a high-level API used for training and evaluation. You can import customized or MindSpore existing metrics as parameters. Models can automatically call the imported metrics for evaluation.
After network model training, metrics need to be used to evaluate the training effect of the network model. Therefore, before specific code is demonstrated, you need to prepare a dataset, load the dataset, and define a simple linear regression network model.
import numpy as np
from mindspore import dataset as ds
def get_data(num, w=2.0, b=3.0):
"""Generate data and corresponding labels."""
for _ in range(num):
x = np.random.uniform(-10.0, 10.0)
noise = np.random.normal(0, 1)
y = x * w + b + noise
yield np.array([x]).astype(np.float32), np.array([y]).astype(np.float32)
def create_dataset(num_data, batch_size=16):
"""Load the dataset."""
dataset = ds.GeneratorDataset(list(get_data(num_data)), column_names=['data', 'label'])
dataset = dataset.batch(batch_size)
return dataset
Using Built-in Evaluation Metrics
When the built-in metrics of MindSpore are transferred to Model as parameters, the metrics can be defined as a dictionary type. The key of the dictionary is a character string, and the value of the dictionary is the built-in evaluation metric of MindSpore. The following example uses nn.Accuracy to compute the classification accuracy.
import mindspore.nn as nn
from mindspore.nn import MAE
from mindspore import Model, LossMonitor
net = nn.Dense(1, 1)
loss_fn = nn.L1Loss()
optimizer = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9)
# Define a model and use the built-in Accuracy function.
model = Model(net, loss_fn, optimizer, metrics={"MAE": MAE()})
train_dataset = create_dataset(num_data=160)
eval_dataset = create_dataset(num_data=160)
train_dataset_size = train_dataset.get_dataset_size()
model.fit(10, train_dataset, eval_dataset, callbacks=LossMonitor(train_dataset_size))
epoch: 1 step: 10, loss is 6.0811052322387695
Eval result: epoch 1, metrics: {'MAE': 5.012505912780762}
epoch: 2 step: 10, loss is 2.7896716594696045
Eval result: epoch 2, metrics: {'MAE': 3.380072832107544}
epoch: 3 step: 10, loss is 3.0297815799713135
Eval result: epoch 3, metrics: {'MAE': 2.5002413272857664}
epoch: 4 step: 10, loss is 2.3680481910705566
Eval result: epoch 4, metrics: {'MAE': 2.4334578275680543}
epoch: 5 step: 10, loss is 1.8126990795135498
Eval result: epoch 5, metrics: {'MAE': 1.8317200541496277}
epoch: 6 step: 10, loss is 1.6006351709365845
Eval result: epoch 6, metrics: {'MAE': 1.521335732936859}
epoch: 7 step: 10, loss is 1.1064929962158203
Eval result: epoch 7, metrics: {'MAE': 1.2528185725212098}
epoch: 8 step: 10, loss is 0.9595810174942017
Eval result: epoch 8, metrics: {'MAE': 1.0719563841819764}
epoch: 9 step: 10, loss is 0.6517931222915649
Eval result: epoch 9, metrics: {'MAE': 0.9766222715377808}
epoch: 10 step: 10, loss is 0.9312882423400879
Eval result: epoch 10, metrics: {'MAE': 0.9238077104091644}
Using Customized Evaluation Metrics
In the following example, the customized evaluation metric MAE() is transferred to Model, and the evaluation dataset is transferred to the model.fit() API for evaluation while training.
The validation result is of the dictionary type. The key of the validation result is the same as that of metrics. The value of the metricsresult is the mean absolute error between the predicted value and the actual value.
train_dataset = create_dataset(num_data=160)
eval_dataset = create_dataset(num_data=160)
model = Model(net, loss_fn, optimizer, metrics={"MAE": MyMAE()})
# Define a model and transfer the customized metrics function MAE to the model.
model.fit(10, train_dataset, eval_dataset, callbacks=LossMonitor(train_dataset_size))
epoch: 1 step: 10, loss is 0.5679571628570557
Eval result: epoch 1, metrics: {'MAE': 0.7907268464565277}
epoch: 2 step: 10, loss is 0.8198273181915283
Eval result: epoch 2, metrics: {'MAE': 0.7729107916355134}
epoch: 3 step: 10, loss is 0.5721814036369324
Eval result: epoch 3, metrics: {'MAE': 0.7661101937294006}
epoch: 4 step: 10, loss is 0.6523740291595459
Eval result: epoch 4, metrics: {'MAE': 0.7704753875732422}
epoch: 5 step: 10, loss is 0.5641313791275024
Eval result: epoch 5, metrics: {'MAE': 0.7609358102083206}
epoch: 6 step: 10, loss is 0.774018406867981
Eval result: epoch 6, metrics: {'MAE': 0.7739883124828338}
epoch: 7 step: 10, loss is 0.7306548357009888
Eval result: epoch 7, metrics: {'MAE': 0.7757290184497834}
epoch: 8 step: 10, loss is 0.667199969291687
Eval result: epoch 8, metrics: {'MAE': 0.7627188444137574}
epoch: 9 step: 10, loss is 0.689708948135376
Eval result: epoch 9, metrics: {'MAE': 0.7673796474933624}
epoch: 10 step: 10, loss is 0.7661054134368896
Eval result: epoch 10, metrics: {'MAE': 0.7654164433479309}