{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 自定义Metrics验证模型推理精度\n", "\n", "`Ascend` `GPU` `CPU` `模型评估`\n", "\n", "[![在线运行](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_modelarts.png)](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9taW5kc3BvcmUtd2Vic2l0ZS5vYnMuY24tbm9ydGgtNC5teWh1YXdlaWNsb3VkLmNvbS9ub3RlYm9vay9tYXN0ZXIvcHJvZ3JhbW1pbmdfZ3VpZGUvemhfY24vbWluZHNwb3JlX3NlbGZfZGVmaW5lX21ldHJpYy5pcHluYg==&imageid=65f636a0-56cf-49df-b941-7d2a07ba8c8c) [![下载Notebook](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_notebook.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r1.6/programming_guide/zh_cn/mindspore_self_define_metric.ipynb) [![下载样例代码](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_download_code.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r1.6/programming_guide/zh_cn/mindspore_self_define_metric.py) [![查看源文件](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r1.6/docs/mindspore/programming_guide/source_zh_cn/self_define_metric.ipynb)\n", "\n", "## 概述\n", "\n", "当训练任务结束，常常需要评价函数(Metrics)来评估模型的好坏。不同的训练任务往往需要不同的Metrics函数。例如，对于二分类问题，常用的评价指标有precision(准确率)、recall(召回率)等， 而对于多分类任务，可使用宏平均(Macro)和微平均(Micro)来评估。MindSpore中提供的Metrics有：`nn.Accuracy`、`nn.Pecision`、`nn.MAE`、`nn.Topk`、`nn.MSE`等，详情可参考：[Metric](https://www.mindspore.cn/docs/api/zh-CN/r1.6/api_python/mindspore.nn.html#metrics) 。虽然MindSpore提供了大部分常见任务的评价指标，但是无法满足所有任务的需求。因此使用者可针对具体的任务自定义Metrics来评估训练的模型。\n", "\n", "以下通过示例来介绍如何自定义Metrics以及如何在`nn.Model`中使用。\n", "\n", "## 自定义Metrics步骤\n", "\n", "所有的metrics都需要继承`nn.Metric`父类，用户只需要重新实现父类中的`clear`、`update`和`eval`即可。其中`clear`用于初始化相关内部参数；`update`接收网络的预测和真值标签，更新内部变量；`eval`用于计算相关指标并返回计算结果。下面我们以简单的`MAE`为例，介绍这三个函数。\n", "\n", "示例代码如下：" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T11:04:40.488396Z", "start_time": "2022-01-04T11:04:38.090036Z" } }, "outputs": [], "source": [ "import numpy as np\n", "import mindspore\n", "from mindspore import Tensor\n", "from mindspore.nn import Metric\n", "from mindspore.nn import rearrange_inputs\n", "\n", "class MyMAE(Metric):\n", "\n", " def __init__(self):\n", " super(MyMAE, self).__init__()\n", " self.clear()\n", "\n", " def clear(self):\n", " \"\"\"Clears the internal evaluation result.\"\"\"\n", " self._abs_error_sum = 0\n", " self._samples_num = 0\n", "\n", " @rearrange_inputs\n", " def update(self, *inputs):\n", " if len(inputs) != 2:\n", " raise ValueError('Mean absolute error need 2 inputs (y_pred, y), but got {}'.format(len(inputs)))\n", " y_pred = self._convert_data(inputs[0])\n", " y = self._convert_data(inputs[1])\n", " abs_error_sum = np.abs(y.reshape(y_pred.shape) - y_pred)\n", " self._abs_error_sum += abs_error_sum.sum()\n", " self._samples_num += y.shape[0]\n", "\n", " def eval(self):\n", " if self._samples_num == 0:\n", " raise RuntimeError('Total samples num must not be 0.')\n", " return self._abs_error_sum / self._samples_num" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- `clear`方法初始化了两个变量，其中`_abs_error_sum`用于保存误差和，`_samples_num`用于累计数据量。\n", "- `update`方法接收网络预测输出和标签，计算误差，并更新`_abs_error_sum`和`_samples_num`。其中`_convert_data`用于将`Tensor`转换为numpy, 后续可利用numpy做相关计算。如果用户评估网络有多个输出，但只用两个输出进行评估，此时可以使用装饰器`rearrange_inputs`中的`set_indexes`方法指定评估网络输出中哪些用于计算评估指标。update一般在每个step进行计算并更新统计值。\n", "- `eval`方法计算最终的指标并返回结果。eval一般在一个epoch结束后计算最终的评估结果。\n", "\n", "下面我们使用上面定义的MAE，计算几组数据的MAE指标。" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T11:04:40.498127Z", "start_time": "2022-01-04T11:04:40.490472Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.1499999612569809\n" ] } ], "source": [ "x = Tensor(np.array([[0.1, 0.2, 0.6, 0.9], [0.1, 0.2, 0.6, 0.9]]), mindspore.float32)\n", "y = Tensor(np.array([[0.1, 0.25, 0.7, 0.9], [0.1, 0.25, 0.7, 0.9]]), mindspore.float32)\n", "error = MyMAE()\n", "error.clear()\n", "error.update(x, y)\n", "result = error.eval()\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": { "ExecuteTime": { "end_time": "2022-01-04T11:03:52.623069Z", "start_time": "2022-01-04T11:03:52.618464Z" } }, "source": [ "下面的例子展示了装饰器`rearrange_inputs`的用法。假设网络共有三个输出，其中第0， 2个输出可用来计算指标。" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T11:04:40.519643Z", "start_time": "2022-01-04T11:04:40.499138Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.1499999612569809\n" ] } ], "source": [ "x = Tensor(np.array([[0.1, 0.2, 0.6, 0.9], [0.1, 0.2, 0.6, 0.9]]), mindspore.float32)\n", "y = Tensor(np.array([[0.1, 0.25, 0.7, 0.9], [0.1, 0.25, 0.7, 0.9]]), mindspore.float32)\n", "z = Tensor(np.array([[0.1, 0.25, 0.7, 0.9], [0.1, 0.25, 0.7, 0.9]]), mindspore.float32)\n", "error = MyMAE().set_indexes([0, 2])\n", "error.clear()\n", "error.update(x, y, z)\n", "result = error.eval()\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 在Model中使用Metrics\n", "\n", "`mindspore.Model`是用于训练和测试的高层API，可以将自定义或MindSpore已有的Metrics作为参数传入，Model能够自动调用传入的Metrics进行评估。`mindspore.Model`的信息，请参考[Model](https://www.mindspore.cn/docs/api/zh-CN/r1.6/api_python/mindspore/mindspore.Model.html#mindspore.Model)。\n", "\n", "```python\n", "model = Model(network, loss, metrics={\"MyMAE\":MyMAE()})\n", "output = model.eval(eval_dataset)\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "MindSpore", "language": "python", "name": "mindspore" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.5" } }, "nbformat": 4, "nbformat_minor": 4 }