{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 使用不确定性估计工具箱\n", "\n", "[![下载Notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.7/resource/_static/logo_notebook.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r1.7/probability/zh_cn/mindspore_using_the_uncertainty_toolbox.ipynb) [![下载样例代码](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.7/resource/_static/logo_download_code.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r1.7/probability/zh_cn/mindspore_using_the_uncertainty_toolbox.py) [![查看源文件](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.7/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r1.7/docs/probability/docs/source_zh_cn/using_the_uncertainty_toolbox.ipynb)\n", "\n", "贝叶斯神经网络的优势之一就是可以获取不确定性，MindSpore Probability在上层提供了不确定性估计的工具箱，用户可以很方便地使用该工具箱计算不确定性。不确定性意味着深度学习模型对预测结果的不确定程度。目前，大多数深度学习算法只能给出预测结果，而不能判断预测结果的可靠性。不确定性主要有两种类型：偶然不确定性和认知不确定性。 \n", "\n", "- 偶然不确定性（Aleatoric Uncertainty）：描述数据中的内在噪声，即无法避免的误差，这个现象不能通过增加采样数据来削弱。\n", "- 认知不确定性（Epistemic Uncertainty）：模型自身对输入数据的估计可能因为训练不佳、训练数据不够等原因而不准确，可以通过增加训练数据等方式来缓解。\n", "\n", "不确定性估计工具箱，适用于主流的深度学习模型，如回归、分类等。在推理阶段，利用不确定性估计工具箱，开发人员只需通过训练模型和训练数据集，指定需要估计的任务和样本，即可得到任意不确定性和认知不确定性。基于不确定性信息，开发人员可以更好地理解模型和数据集。\n", "\n", "本例将使用MNIST数据集和LeNet5网络模型示例，进行本次体验。\n", "\n", "1. 数据准备。\n", "2. 定义深度学习网络。\n", "3. 初始化不确定性评估工具箱。\n", "4. 评估认知不确定性。\n", "\n", "> 本例适用于GPU和Ascend环境，你可以在这里下载完整的样例代码：。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 数据准备\n", "\n", "### 下载数据集\n", "\n", "以下示例代码将MNIST数据集下载并解压到指定位置。" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "import requests\n", "\n", "requests.packages.urllib3.disable_warnings()\n", "\n", "def download_dataset(dataset_url, path):\n", " filename = dataset_url.split(\"/\")[-1]\n", " save_path = os.path.join(path, filename)\n", " if os.path.exists(save_path):\n", " return\n", " if not os.path.exists(path):\n", " os.makedirs(path)\n", " res = requests.get(dataset_url, stream=True, verify=False)\n", " with open(save_path, \"wb\") as f:\n", " for chunk in res.iter_content(chunk_size=512):\n", " if chunk:\n", " f.write(chunk)\n", " print(\"The {} file is downloaded and saved in the path {} after processing\".format(os.path.basename(dataset_url), path))\n", "\n", "train_path = \"datasets/MNIST_Data/train\"\n", "test_path = \"datasets/MNIST_Data/test\"\n", "\n", "download_dataset(\"https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/train-labels-idx1-ubyte\", train_path)\n", "download_dataset(\"https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/train-images-idx3-ubyte\", train_path)\n", "download_dataset(\"https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/t10k-labels-idx1-ubyte\", test_path)\n", "download_dataset(\"https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/t10k-images-idx3-ubyte\", test_path)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "下载的数据集文件的目录结构如下：\n", "\n", "```text\n", "./datasets/MNIST_Data\n", "├── test\n", "│ ├── t10k-images-idx3-ubyte\n", "│ └── t10k-labels-idx1-ubyte\n", "└── train\n", " ├── train-images-idx3-ubyte\n", " └── train-labels-idx1-ubyte\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 数据增强\n", "\n", "定义数据集增强函数，并将原始数据增强为适用于LeNet网络的数据。" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import mindspore.dataset as ds\n", "import mindspore.dataset.transforms.c_transforms as C\n", "import mindspore.dataset.vision.c_transforms as CV\n", "from mindspore.dataset.vision import Inter\n", "\n", "def create_dataset(data_path, batch_size=32, repeat_size=1,\n", " num_parallel_workers=1):\n", " \"\"\"\n", " create dataset for train or test\n", " \"\"\"\n", " # define dataset\n", " mnist_ds = ds.MnistDataset(data_path)\n", "\n", " # define some parameters needed for data enhancement and rough justification\n", " resize_height, resize_width = 32, 32\n", " rescale = 1.0 / 255.0\n", " shift = 0.0\n", " rescale_nml = 1 / 0.3081\n", " shift_nml = -1 * 0.1307 / 0.3081\n", "\n", " # according to the parameters, generate the corresponding data enhancement method\n", " c_trans = [\n", " CV.Resize((resize_height, resize_width), interpolation=Inter.LINEAR),\n", " CV.Rescale(rescale_nml, shift_nml),\n", " CV.Rescale(rescale, shift),\n", " CV.HWC2CHW()\n", " ]\n", " type_cast_op = C.TypeCast(mstype.int32)\n", "\n", " # using map to apply operations to a dataset\n", " mnist_ds = mnist_ds.map(operations=type_cast_op, input_columns=\"label\", num_parallel_workers=num_parallel_workers)\n", " mnist_ds = mnist_ds.map(operations=c_trans, input_columns=\"image\", num_parallel_workers=num_parallel_workers)\n", "\n", " # process the generated dataset\n", " buffer_size = 10000\n", " mnist_ds = mnist_ds.shuffle(buffer_size=buffer_size)\n", " mnist_ds = mnist_ds.batch(batch_size, drop_remainder=True)\n", " mnist_ds = mnist_ds.repeat(repeat_size)\n", "\n", " return mnist_ds" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 定义深度学习网络\n", "\n", "本例采用LeNet5深度神经网络，在MindSpore中实现如下：" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import mindspore.nn as nn\n", "from mindspore import load_checkpoint, load_param_into_net\n", "from mindspore.common.initializer import Normal\n", "\n", "class LeNet5(nn.Cell):\n", " \"\"\"Lenet network structure.\"\"\"\n", " # define the operator required\n", " def __init__(self, num_class=10, num_channel=1):\n", " super(LeNet5, self).__init__()\n", " self.conv1 = nn.Conv2d(num_channel, 6, 5, pad_mode='valid')\n", " self.conv2 = nn.Conv2d(6, 16, 5, pad_mode='valid')\n", " self.fc1 = nn.Dense(16 * 5 * 5, 120, weight_init=Normal(0.02))\n", " self.fc2 = nn.Dense(120, 84, weight_init=Normal(0.02))\n", " self.fc3 = nn.Dense(84, num_class, weight_init=Normal(0.02))\n", " self.relu = nn.ReLU()\n", " self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)\n", " self.flatten = nn.Flatten()\n", "\n", " # use the preceding operators to construct networks\n", " def construct(self, x):\n", " x = self.max_pool2d(self.relu(self.conv1(x)))\n", " x = self.max_pool2d(self.relu(self.conv2(x)))\n", " x = self.flatten(x)\n", " x = self.relu(self.fc1(x))\n", " x = self.relu(self.fc2(x))\n", " x = self.fc3(x)\n", " return x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 初始化不确定性工具箱\n", "\n", "初始化不确定性工具箱的`UncertaintyEvaluation`功能，准备如下：\n", "\n", "1. 准备模型权重参数文件。\n", "2. 将模型权重参数文件载入神经网络中。\n", "3. 将训练数据集增强为适用于神经网络的数据。\n", "4. 将上述网络和数据集载入到`UncertaintyEvaluation`中。\n", "\n", "MindSpore中使用不确定性工具箱`UncertaintyEvaluation`接口来测量模型偶然不确定性和认知不确定性，更多使用方法请参见[API](https://www.mindspore.cn/docs/zh-CN/r1.7/api_python/mindspore.nn.probability.html#mindspore-nn-probability)。\n", "\n", "### 准备模型权重参数文件\n", "\n", "本例已经准备好了对应的模型权重参数文件`checkpoint_lenet.ckpt`，本参数文件为[初学入门](https://www.mindspore.cn/tutorials/zh-CN/r1.7/beginner/quick_start.html)中训练完成5个epoch后保存的权重参数文件，执行如下命令进行下载：" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "download_dataset(\"https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/models/checkpoint_lenet.ckpt\", \".\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 完成初始化\n", "\n", "将需要进行不确定性测量的DNN网络与训练数据集载入，由于不确定性测量需要贝叶斯网络，所以当第一次调用初始化完成的不确定性测量工具时，会将DNN网络转成贝叶斯网络进行训练，完成后可传入对应的数据进行偶然不确定性或认知不确定性进行测量。" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "scrolled": true }, "outputs": [], "source": [ "from mindspore import context\n", "from mindspore.nn.probability.toolbox import UncertaintyEvaluation\n", "from mindspore import dtype as mstype\n", "\n", "context.set_context(mode=context.GRAPH_MODE, device_target=\"GPU\")\n", "# get trained model\n", "network = LeNet5()\n", "param_dict = load_checkpoint('checkpoint_lenet.ckpt')\n", "load_param_into_net(network, param_dict)\n", "# get train\n", "ds_train = create_dataset('./datasets/MNIST_Data/train')\n", "evaluation = UncertaintyEvaluation(model=network,\n", " train_dataset=ds_train,\n", " task_type='classification',\n", " num_classes=10,\n", " epochs=1,\n", " epi_uncer_model_path=None,\n", " ale_uncer_model_path=None,\n", " save_model=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 评估认知不确定性\n", "\n", "### 转换成贝叶斯训练测量\n", "\n", "首先将验证数据集取出一个`batch`，进行认知不确定性测量，首次调用时会将原本深度神经网络转换为贝叶斯网络进行训练。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "epoch: 1 step: 1, loss is 0.14702837\n", "epoch: 1 step: 2, loss is 0.00017862688\n", "epoch: 1 step: 3, loss is 0.09421586\n", "epoch: 1 step: 4, loss is 0.0003434865\n", "epoch: 1 step: 5, loss is 7.1358285e-05\n", "... ...\n", "epoch: 1 step: 1871, loss is 0.20069705\n", "epoch: 1 step: 1872, loss is 0.12135945\n", "epoch: 1 step: 1873, loss is 0.04572148\n", "epoch: 1 step: 1874, loss is 0.04962858\n", "epoch: 1 step: 1875, loss is 0.0019944885\n" ] } ], "source": [ "ds_test = create_dataset(\"./datasets/MNIST_Data/test\")\n", "batch_data = next(ds_test.create_dict_iterator())\n", "eval_images = batch_data[\"image\"]\n", "eval_labels = batch_data[\"label\"]\n", "epistemic_uncertainty = evaluation.eval_epistemic_uncertainty(eval_images)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`evaluation.eval_epistemic_uncertainty`：认知不确定性测量接口，第一次调用时会使用训练数据对DNN模型进行转换成贝叶斯训练。 \n", "`eval_images`：即偶然不确定性测试使用的`batch`图片。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 打印认知不确定性" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "取一个batch的数据将label打印出来。" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2 9 4 3 9 9 2 4 9 6 0 5 6 8 7 6 1 9 7 6 5 4 0 3 7 7 6 7 7 4 6 2]\n", "(32, 10)\n" ] } ], "source": [ "print(eval_labels)\n", "print(epistemic_uncertainty.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "认知不确定性内容为32张图片对应0-9的分类模型的不确定性值。\n", "\n", "取前面两个图片打印出对应模型的的偶然不确定性值。" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "the picture one, number is 2, epistemic uncertainty is:\n", "[0.75372726 0.2053496 3.737096 0.7113453 0.93452704 0.40339947\n", " 0.91918266 0.44237098 0.40863538 0.8195221 ]\n", "the picture two, number is 9, epistemic uncertainty is:\n", "[0.97602427 0.37808532 0.4955423 0.17907992 1.3365419 0.20227651\n", " 2.2211757 0.27501273 0.30733848 3.7536747 ]\n" ] } ], "source": [ "print(\"the picture one, number is {}, epistemic uncertainty is:\\n{}\".format(eval_labels[0], epistemic_uncertainty[0]))\n", "print(\"the picture two, number is {}, epistemic uncertainty is:\\n{}\".format(eval_labels[1], epistemic_uncertainty[1]))" ] } ], "metadata": { "kernelspec": { "display_name": "MindSpore", "language": "python", "name": "mindspore" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }