{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 评价指标\n", "\n", "[![下载Notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/resource/_static/logo_notebook.png)](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r1.8/tutorials/zh_cn/advanced/train/mindspore_metric.ipynb) [![下载样例代码](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/resource/_static/logo_download_code.png)](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r1.8/tutorials/zh_cn/advanced/train/mindspore_metric.py) [![查看源文件](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r1.8/tutorials/source_zh_cn/advanced/train/metric.ipynb)\n", "\n", "当训练任务结束，常常需要评价函数（Metrics）来评估模型的好坏。不同的训练任务往往需要不同的Metrics函数。例如，对于二分类问题，常用的评价指标有precision（准确率）、recall（召回率）等，而对于多分类任务，可使用宏平均（Macro）和微平均（Micro）来评估。\n", "\n", "MindSpore提供了大部分常见任务的评价函数，如`nn.Accuracy`、`nn.Precision`、`nn.MAE`和`nn.MSE`等，由于MindSpore提供的评价函数无法满足所有任务的需求，很多情况下用户需要针对具体的任务自定义Metrics来评估训练的模型。\n", "\n", "本章主要介绍如何自定义Metrics以及如何在`nn.Model`中使用Metrics。\n", "\n", "> 详情可参考[评价指标](https://www.mindspore.cn/docs/zh-CN/r1.8/api_python/mindspore.nn.html#评价指标)。\n", "\n", "## 自定义Metrics\n", "\n", "自定义Metrics函数需要继承`nn.Metric`父类，并重新实现父类中的`clear`方法、`update`方法和`eval`方法。\n", "\n", "- `clear`：初始化相关的内部参数。\n", "- `update`：接收网络预测输出和标签，计算误差，每次step后并更新内部评估结果。\n", "- `eval`：计算最终评估结果，在没次epoch结束后计算最终的评估结果。\n", "\n", "平均绝对误差（MAE）算法如式(1)所示：\n", "\n", "$$ MAE=\\frac{1}{n}\\sum_{i=1}^n\\lvert ypred_i - y_i \\rvert \\tag{1}$$\n", "\n", "下面以简单的MAE算法为例，介绍`clear`、`update`和`eval`三个函数及其使用方法。" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T11:04:40.488396Z", "start_time": "2022-01-04T11:04:38.090036Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "output(y_pred, y): 0.1499999612569809\n" ] } ], "source": [ "import numpy as np\n", "import mindspore as ms\n", "import mindspore.nn as nn\n", "\n", "class MyMAE(nn.Metric):\n", " def __init__(self):\n", " super(MyMAE, self).__init__()\n", " self.clear()\n", "\n", " def clear(self):\n", " \"\"\"初始化变量_abs_error_sum和_samples_num\"\"\"\n", " self._abs_error_sum = 0 # 保存误差和\n", " self._samples_num = 0 # 累计数据量\n", "\n", " @nn.rearrange_inputs\n", " def update(self, *inputs):\n", " \"\"\"更新_abs_error_sum和_samples_num\"\"\"\n", " y_pred = inputs[0].asnumpy()\n", " y = inputs[1].asnumpy()\n", "\n", " # 计算预测值与真实值的绝对误差\n", " abs_error_sum = np.abs(y - y_pred)\n", " self._abs_error_sum += abs_error_sum.sum()\n", "\n", " # 样本的总数\n", " self._samples_num += y.shape[0]\n", "\n", " def eval(self):\n", " \"\"\"计算最终评估结果\"\"\"\n", " return self._abs_error_sum / self._samples_num\n", "\n", "# 网络有两个输出\n", "y_pred = ms.Tensor(np.array([[0.1, 0.2, 0.6, 0.9], [0.1, 0.2, 0.6, 0.9]]), ms.float32)\n", "y = ms.Tensor(np.array([[0.1, 0.25, 0.7, 0.9], [0.1, 0.25, 0.7, 0.9]]), ms.float32)\n", "\n", "error = MyMAE()\n", "error.clear()\n", "error.update(y_pred, y)\n", "result = error.eval()\n", "print(\"output(y_pred, y):\", result)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "值得注意的是，`update`中如果用户评估网络有多个输出，但只用两个输出进行评估，此时可以使用`set_indexes`方法重排`update`的输入用于计算评估指标。使用`set_indexes`方法，需要用装饰器`nn.rearrange_inputs`修饰`update`方法，否则使用set_indexes配置的输入不生效。" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "output(y_pred,z): 0.24999992549419403\n" ] } ], "source": [ "# 网络有三个输出:y_pred，y，z\n", "z = ms.Tensor(np.array([[0.1, 0.25, 0.7, 0.8], [0.1, 0.25, 0.7, 0.8]]), ms.float32)\n", "\n", "# 设置使用y_pred,z进行评估\n", "error = MyMAE().set_indexes([0, 2])\n", "error.clear()\n", "error.update(y_pred, y, z)\n", "result = error.eval()\n", "print(\"output(y_pred,z):\", result)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 模型训练中使用Metrics\n", "\n", "[mindspore.Model](https://www.mindspore.cn/docs/zh-CN/r1.8/api_python/mindspore/mindspore.Model.html#mindspore.Model)是用于训练和评估的高层API，可以将自定义或MindSpore已有的Metrics作为参数传入，Model能够自动调用传入的Metrics进行评估。\n", "\n", "在网络模型训练后，需要使用评价指标，来评估网络模型的训练效果，因此在演示具体代码之前首先简单拟定数据集，对数据集进行加载和定义一个简单的线性回归网络模型：\n", "\n", "$$f(x)=w*x+b \\tag{2}$$" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import mindspore.nn as nn\n", "import mindspore as ms\n", "from mindspore import dataset as ds\n", "from mindspore.common.initializer import Normal\n", "\n", "def get_data(num, w=2.0, b=3.0):\n", " \"\"\"生成数据及对应标签\"\"\"\n", " for _ in range(num):\n", " x = np.random.uniform(-10.0, 10.0)\n", " noise = np.random.normal(0, 1)\n", " y = x * w + b + noise\n", " yield np.array([x]).astype(np.float32), np.array([y]).astype(np.float32)\n", "\n", "def create_dataset(num_data, batch_size=16):\n", " \"\"\"加载数据集\"\"\"\n", " dataset = ds.GeneratorDataset(list(get_data(num_data)), column_names=['data', 'label'])\n", " dataset = dataset.batch(batch_size)\n", " return dataset\n", "\n", "class LinearNet(nn.Cell):\n", " \"\"\"定义线性回归网络\"\"\"\n", " def __init__(self):\n", " super(LinearNet, self).__init__()\n", " self.fc = nn.Dense(1, 1, Normal(0.02), Normal(0.02))\n", "\n", " def construct(self, x):\n", " return self.fc(x)\n", "\n", "loss = nn.L1Loss()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 使用内置评价指标\n", "\n", "使用MindSpore内置的Metrics作为参数传入Model时，Metrics可以定义为一个字典类型，字典的key值为字符串类型，字典的value值为MindSpore内置的[评价指标](https://www.mindspore.cn/docs/zh-CN/r1.8/api_python/mindspore.nn.html#评价指标)，如下示例使用`nn.Accuracy`计算分类的准确率。" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch:[ 0/ 1], step:[ 1/ 10], loss:[10.206/10.206], time:148.661 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 2/ 10], loss:[8.827/9.516], time:0.671 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 3/ 10], loss:[13.232/10.755], time:0.681 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 4/ 10], loss:[10.893/10.789], time:0.704 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 5/ 10], loss:[8.339/10.299], time:0.668 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 6/ 10], loss:[8.881/10.063], time:0.826 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 7/ 10], loss:[6.288/9.524], time:0.923 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 8/ 10], loss:[8.166/9.354], time:0.932 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 9/ 10], loss:[7.538/9.152], time:0.932 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 10/ 10], loss:[5.517/8.789], time:0.980 ms, lr:0.00500\n", "Epoch time: 167.900 ms, per step time: 16.790 ms, avg loss: 8.789\n", "{'MAE': 5.931522464752197}\n" ] } ], "source": [ "import mindspore.nn as nn\n", "import mindspore as ms\n", "from mindvision.engine.callback import LossMonitor\n", "\n", "ds_train = create_dataset(num_data=160)\n", "net = LinearNet()\n", "opt = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9)\n", "\n", "# 定义model，使用内置的Accuracy函数\n", "model = ms.Model(net, loss, opt, metrics={\"MAE\": nn.MAE()})\n", "model.train(epoch=1, train_dataset=ds_train, callbacks=LossMonitor(0.005))\n", "\n", "# 模型评估\n", "ds_eval = create_dataset(num_data=160)\n", "output = model.eval(ds_eval)\n", "print(output)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 使用自定义评价指标\n", "\n", "如下示例在`Model`中传入上述自定义的评估指标`MAE()`，将验证数据集传入`model.eval()`接口进行验证。\n", "\n", "验证结果为一个字典类型，验证结果的key值与`metrics`的key值相同，验证结果的value值为预测值与实际值的平均绝对误差。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch:[ 0/ 1], step:[ 1/ 10], loss:[9.931/9.931], time:157.518 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 2/ 10], loss:[10.705/10.318], time:0.751 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 3/ 10], loss:[11.313/10.650], time:0.722 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 4/ 10], loss:[9.445/10.349], time:0.738 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 5/ 10], loss:[5.492/9.377], time:0.737 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 6/ 10], loss:[8.060/9.158], time:0.839 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 7/ 10], loss:[7.866/8.973], time:0.900 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 8/ 10], loss:[7.264/8.760], time:0.863 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 9/ 10], loss:[8.975/8.784], time:0.885 ms, lr:0.00500\n", "Epoch:[ 0/ 1], step:[ 10/ 10], loss:[7.630/8.668], time:0.958 ms, lr:0.00500\n", "Epoch time: 177.346 ms, per step time: 17.735 ms, avg loss: 8.668\n", "{'MAE': 5.533915233612061}\n" ] } ], "source": [ "ds_train = create_dataset(num_data=160)\n", "net1 = LinearNet()\n", "opt = nn.Momentum(net1.trainable_params(), learning_rate=0.005, momentum=0.9)\n", "\n", "# 定义model，将自定义metrics函数MAE传入Model中\n", "model1 = ms.Model(net1, loss, opt, metrics={\"MAE\": MyMAE()})\n", "model1.train(epoch=1, train_dataset=ds_train, callbacks=LossMonitor(0.005))\n", "\n", "# 模型评估\n", "ds_eval = create_dataset(num_data=160)\n", "output = model1.eval(ds_eval)\n", "print(output)" ] } ], "metadata": { "kernelspec": { "display_name": "MindSpore", "language": "python", "name": "mindspore" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.0" } }, "nbformat": 4, "nbformat_minor": 4 }