{
"cells": [
{
"cell_type": "markdown",
"id": "f2df123f-eb9a-4816-8cef-2fa48f21cffd",
"metadata": {},
"source": [
"[](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9taW5kc3BvcmUtd2Vic2l0ZS5vYnMuY24tbm9ydGgtNC5teWh1YXdlaWNsb3VkLmNvbS9ub3RlYm9vay9yMi4yL3R1dG9yaWFscy96aF9jbi9hZHZhbmNlZC9tb2RlbC9taW5kc3BvcmVfY2FsbGJhY2suaXB5bmI=&imageid=4c43b3ad-9df7-4b83-a096-c775dc4ba243) [](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r2.2/tutorials/zh_cn/advanced/model/mindspore_callback.ipynb) \n",
"[](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r2.2/tutorials/zh_cn/advanced/model/mindspore_callback.py) \n",
"[](https://gitee.com/mindspore/docs/blob/r2.2/tutorials/source_zh_cn/advanced/model/callback.ipynb)"
]
},
{
"cell_type": "markdown",
"id": "7bcf904e",
"metadata": {},
"source": [
"# 回调机制 Callback\n",
"\n",
"在深度学习训练过程中，为及时掌握网络模型的训练状态、实时观察网络模型各参数的变化情况和实现训练过程中用户自定义的一些操作，MindSpore提供了回调机制（Callback）来实现上述功能。\n",
"\n",
"Callback回调机制一般用在网络模型训练过程`Model.train`中，MindSpore的`Model`会按照Callback列表`callbacks`顺序执行回调函数，用户可以通过设置不同的回调类来实现在训练过程中或者训练后执行的功能。\n",
"\n",
"> 更多内置回调类的信息及使用方式请参考[API文档](https://www.mindspore.cn/docs/zh-CN/r2.2/api_python/train/mindspore.train.Callback.html#mindspore.train.Callback)。\n",
"\n",
"## Callback介绍\n",
"\n",
"当聊到回调Callback的时候，大部分用户都会觉得很难理解，是不是需要堆栈或者特殊的调度方式，实际上我们简单的理解回调：\n",
"\n",
"假设函数A有一个参数，这个参数是个函数B，当函数A执行完以后执行函数B，那么这个过程就叫回调。\n",
"\n",
"`Callback`是回调的意思，MindSpore中的回调函数实际上不是一个函数而是一个类，用户可以使用回调机制来**观察训练过程中网络内部的状态和相关信息，或在特定时期执行特定动作**。\n",
"\n",
"例如监控损失函数Loss、保存模型参数ckpt、动态调整参数lr、提前终止训练任务等。下面我们继续以手写体识别模型为例，介绍常见的内置回调函数和自定义回调函数。"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "5ef29213-fb5d-4887-b56a-2e9758adf173",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading data from https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/datasets/MNIST_Data.zip (10.3 MB)\n",
"\n",
"file_sizes: 100%|██████████████████████████| 10.8M/10.8M [00:01<00:00, 10.0MB/s]\n",
"Extracting zip file...\n",
"Successfully downloaded / unzipped to ./\n"
]
}
],
"source": [
"import mindspore\n",
"from mindspore import nn\n",
"from mindspore.dataset import vision, transforms\n",
"from mindspore.dataset import MnistDataset\n",
"from mindspore.train import Model\n",
"\n",
"# Download data from open datasets\n",
"from download import download\n",
"\n",
"url = \"https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/\" \\\n",
" \"notebook/datasets/MNIST_Data.zip\"\n",
"path = download(url, \"./\", kind=\"zip\", replace=True)\n",
"\n",
"\n",
"def datapipe(path, batch_size):\n",
" image_transforms = [\n",
" vision.Rescale(1.0 / 255.0, 0),\n",
" vision.Normalize(mean=(0.1307,), std=(0.3081,)),\n",
" vision.HWC2CHW()\n",
" ]\n",
" label_transform = transforms.TypeCast(mindspore.int32)\n",
"\n",
" dataset = MnistDataset(path)\n",
" dataset = dataset.map(image_transforms, 'image')\n",
" dataset = dataset.map(label_transform, 'label')\n",
" dataset = dataset.batch(batch_size)\n",
" return dataset\n",
"\n",
"# Define model\n",
"class Network(nn.Cell):\n",
" def __init__(self):\n",
" super().__init__()\n",
" self.flatten = nn.Flatten()\n",
" self.dense_relu_sequential = nn.SequentialCell(\n",
" nn.Dense(28*28, 512),\n",
" nn.ReLU(),\n",
" nn.Dense(512, 512),\n",
" nn.ReLU(),\n",
" nn.Dense(512, 10)\n",
" )\n",
"\n",
" def construct(self, x):\n",
" x = self.flatten(x)\n",
" logits = self.dense_relu_sequential(x)\n",
" return logits\n",
"\n",
"model = Network()\n",
"loss_fn = nn.CrossEntropyLoss()\n",
"optimizer = nn.SGD(model.trainable_params(), 1e-2)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "48b9deea-8766-4b8f-8d3a-f8d1d7762381",
"metadata": {},
"outputs": [],
"source": [
"train_dataset = datapipe('MNIST_Data/train', 64)\n",
"test_dataset = datapipe('MNIST_Data/test', 64)\n",
"\n",
"trainer = Model(model, loss_fn=loss_fn, optimizer=optimizer, metrics={'accuracy'})"
]
},
{
"cell_type": "markdown",
"id": "a31c0fc7",
"metadata": {},
"source": [
"## 常用的内置回调函数\n",
"\n",
"MindSpore提供`Callback`能力，支持用户在训练/推理的特定阶段，插入自定义的操作。\n",
"\n",
"### ModelCheckpoint\n",
"\n",
"用于保存训练后的网络模型和参数，方便进行再推理或再训练，MindSpore提供了[ModelCheckpoint](https://mindspore.cn/docs/zh-CN/r2.2/api_python/train/mindspore.train.ModelCheckpoint.html#mindspore.train.ModelCheckpoint)接口，一般与配置保存信息接口[CheckpointConfig](https://mindspore.cn/docs/zh-CN/r2.2/api_python/train/mindspore.train.CheckpointConfig.html#mindspore.train.CheckpointConfig)配合使用。"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "8da633ba",
"metadata": {},
"outputs": [],
"source": [
"from mindspore.train import CheckpointConfig, ModelCheckpoint\n",
"\n",
"# 设置保存模型的配置信息\n",
"config = CheckpointConfig(save_checkpoint_steps=1875, keep_checkpoint_max=10)\n",
"# 实例化保存模型回调接口，定义保存路径和前缀名\n",
"ckpt_callback = ModelCheckpoint(prefix=\"mnist\", directory=\"./checkpoint\", config=config)\n",
"\n",
"# 开始训练，加载保存模型和参数回调函数\n",
"trainer.train(1, train_dataset, callbacks=[ckpt_callback])"
]
},
{
"cell_type": "markdown",
"id": "fa304ea4",
"metadata": {},
"source": [
"上面代码运行后，生成的Checkpoint文件目录结构如下：\n",
"\n",
"```text\n",
"./checkpoint/\n",
"├── mnist-1_938.ckpt # 保存参数文件\n",
"└── mnist-graph.meta # 编译后的计算图\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "0e9e05b2-2787-4655-828f-128ed95c9589",
"metadata": {},
"source": [
"### LossMonitor\n",
"\n",
"用于监控训练或测试过程中的损失函数值Loss变化情况，可设置`per_print_times`控制打印Loss值的间隔。"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "a100fa97",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"epoch: 1 step: 300, loss is 0.45305341482162476\n",
"epoch: 1 step: 600, loss is 0.2915695905685425\n",
"epoch: 1 step: 900, loss is 0.5174192190170288\n"
]
}
],
"source": [
"from mindspore.train import LossMonitor\n",
"\n",
"loss_monitor = LossMonitor(300)\n",
"# 开始训练，加载保存模型和参数回调函数，LossMonitor的入参0.01为学习率，375为步长\n",
"trainer.train(1, train_dataset, callbacks=[loss_monitor])"
]
},
{
"cell_type": "markdown",
"id": "5d9d3a68-8a78-4d83-9d2d-f78945cb903a",
"metadata": {},
"source": [
"训练场景下，LossMonitor监控训练的Loss值；边训练边推理场景下，监控训练的Loss值和推理的Metrics值。"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "c917cd7e-5ef8-4172-b9f8-a486ea20597d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"epoch: 1 step: 300, loss is 0.3167177438735962\n",
"epoch: 1 step: 600, loss is 0.36215940117836\n",
"epoch: 1 step: 900, loss is 0.25714176893234253\n",
"Eval result: epoch 1, metrics: {'accuracy': 0.9202}\n"
]
}
],
"source": [
"trainer.fit(1, train_dataset, test_dataset, callbacks=[loss_monitor])"
]
},
{
"cell_type": "markdown",
"id": "a0363b25-b05a-4640-b159-d8fc05049590",
"metadata": {},
"source": [
"### TimeMonitor\n",
"\n",
"用于监控训练或测试过程的执行时间。可设置`data_size`控制打印执行时间的间隔。"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "3d5cfe4b-57e8-4c5d-8ef4-5447be15fbe3",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Train epoch time: 7388.254 ms, per step time: 7.877 ms\n"
]
}
],
"source": [
"from mindspore.train import TimeMonitor\n",
"\n",
"time_monitor = TimeMonitor()\n",
"trainer.train(1, train_dataset, callbacks=[time_monitor])"
]
},
{
"cell_type": "markdown",
"id": "7add14c8",
"metadata": {},
"source": [
"## 自定义回调机制\n",
"\n",
"MindSpore不仅有功能强大的内置回调函数，当用户有自己的特殊需求时，还可以基于`Callback`基类自定义回调类。\n",
"\n",
"用户可以基于`Callback`基类，根据自身的需求，实现自定义`Callback`。`Callback`基类定义如下所示："
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "c942ff83",
"metadata": {},
"outputs": [],
"source": [
"class Callback():\n",
" \"\"\"Callback base class\"\"\"\n",
" def on_train_begin(self, run_context):\n",
" \"\"\"Called once before the network executing.\"\"\"\n",
"\n",
" def on_train_epoch_begin(self, run_context):\n",
" \"\"\"Called before each epoch beginning.\"\"\"\n",
"\n",
" def on_train_epoch_end(self, run_context):\n",
" \"\"\"Called after each epoch finished.\"\"\"\n",
"\n",
" def on_train_step_begin(self, run_context):\n",
" \"\"\"Called before each step beginning.\"\"\"\n",
"\n",
" def on_train_step_end(self, run_context):\n",
" \"\"\"Called after each step finished.\"\"\"\n",
"\n",
" def on_train_end(self, run_context):\n",
" \"\"\"Called once after network training.\"\"\""
]
},
{
"cell_type": "markdown",
"id": "a18f2897",
"metadata": {},
"source": [
"回调机制可以把训练过程中的重要信息记录下来，通过把一个字典类型变量`RunContext.original_args()`，传递给Callback对象，使得用户可以在各个自定义的Callback中获取到相关属性，执行自定义操作，也可以自定义其他变量传递给`RunContext.original_args()`对象。\n",
"\n",
"`RunContext.original_args()`中的常用属性有：\n",
"\n",
"- epoch_num：训练的epoch的数量\n",
"- batch_num：一个epoch中step的数量\n",
"- cur_epoch_num：当前的epoch数\n",
"- cur_step_num：当前的step数\n",
"\n",
"- loss_fn：损失函数\n",
"- optimizer：优化器\n",
"- train_network：训练的网络\n",
"- train_dataset：训练的数据集\n",
"- net_outputs：网络的输出结果\n",
"\n",
"- parallel_mode：并行模式\n",
"- list_callback：所有的Callback函数\n",
"\n",
"通过下面两个场景，我们可以增加对自定义Callback回调机制功能的了解。\n",
"\n",
"### 自定义终止训练\n",
"\n",
"实现在规定时间内终止训练功能。用户可以设定时间阈值，当训练时间达到这个阈值后就终止训练过程。\n",
"\n",
"下面代码中，通过`run_context.original_args`方法可以获取到`cb_params`字典，字典里会包含前文描述的主要属性信息。\n",
"\n",
"同时可以对字典内的值进行修改和添加，在`begin`函数中定义一个`init_time`对象传递给`cb_params`字典。每个数据迭代结束`step_end`之后会进行判断，当训练时间大于设置的时间阈值时，会向run_context传递终止训练的信号，提前终止训练，并打印当前的epoch、step、loss的值。"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "a3f2b5e8",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Begin training, time is: 1665892816.363511\n",
"End training, time: 1665892820.3696215, epoch: 1, step: 575, loss:Tensor(shape=[], dtype=Float32, value= 0.35758)\n"
]
}
],
"source": [
"import time\n",
"import mindspore as ms\n",
"\n",
"class StopTimeMonitor(ms.train.Callback):\n",
"\n",
" def __init__(self, run_time):\n",
" \"\"\"定义初始化过程\"\"\"\n",
" super(StopTimeMonitor, self).__init__()\n",
" self.run_time = run_time # 定义执行时间\n",
"\n",
" def on_train_begin(self, run_context):\n",
" \"\"\"开始训练时的操作\"\"\"\n",
" cb_params = run_context.original_args()\n",
" cb_params.init_time = time.time() # 获取当前时间戳作为开始训练时间\n",
" print(f\"Begin training, time is: {cb_params.init_time}\")\n",
"\n",
" def on_train_step_end(self, run_context):\n",
" \"\"\"每个step结束后执行的操作\"\"\"\n",
" cb_params = run_context.original_args()\n",
" epoch_num = cb_params.cur_epoch_num # 获取epoch值\n",
" step_num = cb_params.cur_step_num # 获取step值\n",
" loss = cb_params.net_outputs # 获取损失值loss\n",
" cur_time = time.time() # 获取当前时间戳\n",
"\n",
" if (cur_time - cb_params.init_time) > self.run_time:\n",
" print(f\"End training, time: {cur_time}, epoch: {epoch_num}, step: {step_num}, loss:{loss}\")\n",
" run_context.request_stop() # 停止训练\n",
"\n",
"datasize = train_dataset.get_dataset_size()\n",
"trainer.train(5, train_dataset, callbacks=[LossMonitor(datasize), StopTimeMonitor(4)])"
]
},
{
"cell_type": "markdown",
"id": "0f864397",
"metadata": {},
"source": [
"从上面的打印结果可以看出，当第3个epoch的第4673个step执行完时，运行时间到达了阈值并结束了训练。\n",
"\n",
"### 自定义阈值保存模型\n",
"\n",
"该回调机制实现当loss小于设定的阈值时，保存网络模型权重ckpt文件。\n",
"\n",
"示例代码如下："
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "f494e1d8",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Saved checkpoint, loss:0.0390485, current step num: 154.\n",
"Saved checkpoint, loss:0.0481475, current step num: 234.\n",
"Saved checkpoint, loss:0.0477566, current step num: 361.\n",
"Saved checkpoint, loss:0.0314977, current step num: 444.\n",
"Saved checkpoint, loss:0.0463577, current step num: 513.\n",
"Saved checkpoint, loss:0.0408403, current step num: 764.\n",
"Saved checkpoint, loss:0.0308827, current step num: 899.\n"
]
}
],
"source": [
"import mindspore as ms\n",
"\n",
"# 定义保存ckpt文件的回调接口\n",
"class SaveCkptMonitor(ms.train.Callback):\n",
" \"\"\"定义初始化过程\"\"\"\n",
"\n",
" def __init__(self, loss):\n",
" super(SaveCkptMonitor, self).__init__()\n",
" self.loss = loss # 定义损失值阈值\n",
"\n",
" def on_train_step_end(self, run_context):\n",
" \"\"\"定义step结束时的执行操作\"\"\"\n",
" cb_params = run_context.original_args()\n",
" cur_loss = cb_params.net_outputs.asnumpy() # 获取当前损失值\n",
"\n",
" # 如果当前损失值小于设定的阈值就停止训练\n",
" if cur_loss < self.loss:\n",
" # 自定义保存文件名\n",
" file_name = f\"./checkpoint/{cb_params.cur_epoch_num}_{cb_params.cur_step_num}.ckpt\"\n",
" # 保存网络模型\n",
" ms.save_checkpoint(save_obj=cb_params.train_network, ckpt_file_name=file_name)\n",
" print(\"Saved checkpoint, loss:{:8.7f}, current step num:{:4}.\".format(cur_loss, cb_params.cur_step_num))\n",
"\n",
"trainer.train(1, train_dataset, callbacks=[SaveCkptMonitor(0.05)])"
]
},
{
"cell_type": "markdown",
"id": "e85a7eeb",
"metadata": {},
"source": [
"保存目录结构如下：\n",
"\n",
"```text\n",
"./checkpoint/\n",
"├── 1_154.ckpt\n",
"├── 1_234.ckpt\n",
"├── 1_361.ckpt\n",
"├── 1_444.ckpt\n",
"├── 1_513.ckpt\n",
"├── 1_764.ckpt\n",
"├── 1_899.ckpt\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "MindSpore",
"language": "python",
"name": "mindspore"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.5"
}
},
"nbformat": 4,
"nbformat_minor": 5
}