{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 动态图与静态图\n", "\n", "[![](https://gitee.com/mindspore/docs/raw/r1.3/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r1.3/tutorials/source_zh_cn/intermediate/pynative_mode_and_graph_mode.ipynb) [![](https://gitee.com/mindspore/docs/raw/r1.3/resource/_static/logo_notebook.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r1.3/tutorials/zh_cn/mindspore_pynative_mode_and_graph_mode.ipynb) [![](https://gitee.com/mindspore/docs/raw/r1.3/resource/_static/logo_modelarts.png)](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9taW5kc3BvcmUtd2Vic2l0ZS5vYnMuY24tbm9ydGgtNC5teWh1YXdlaWNsb3VkLmNvbS9ub3RlYm9vay9yMS4zL3R1dG9yaWFscy96aF9jbi9taW5kc3BvcmVfcHluYXRpdmVfbW9kZV9hbmRfZ3JhcGhfbW9kZS5pcHluYg==&imageid=59a6e9f5-93c0-44dd-85b0-82f390c5d53b)\n", "\n", "## 概述\n", "\n", "MindSpore支持两种运行模式：\n", "\n", "- Graph模式：静态图模式或者图模式，将神经网络模型编译成一整张图，然后下发执行。该模式利用图优化等技术提高运行性能，同时有助于规模部署和跨平台运行。\n", "- PyNative模式：动态图模式，将神经网络中的各个算子逐一下发执行，方便用户编写和调试神经网络模型。\n", "\n", "默认情况下，MindSpore处于Graph模式，可以通过`context.set_context(mode=context.PYNATIVE_MODE)`切换为PyNative模式；同样地，MindSpore处于PyNative模式时，可以通过 `context.set_context(mode=context.GRAPH_MODE)`切换为Graph模式。\n", "\n", "Graph和PyNative两种模式的区别主要有：\n", "\n", "- 使用场景：Graph模式需要一开始就构建好网络结构，然后框架做整图优化和执行，比较适合网络固定没有变化，且需要高性能的场景。而PyNative模式逐行执行算子，支持单独求梯度。\n", "\n", "- 网络执行：Graph模式和PyNative模式在执行相同的网络和算子时，精度效果是一致的。由于Graph模式运用了图优化、计算图整图下沉等技术，Graph模式执行网络的性能和效率更高。\n", "\n", "- 代码调试：在脚本开发和网络流程调试中，推荐使用PyNative模式进行调试。在PyNative模式下，可以方便地设置断点，获取网络执行的中间结果，也可以通过pdb的方式对网络进行调试。而Graph模式无法设置断点，只能先指定算子进行打印，然后在网络执行完成后查看输出结果。\n", "\n", "下面以Graph模式为例，演示MindSpore单算子、普通函数、模型的执行方式，并进一步说明如何在PyNative模式下进行性能改进及梯度求取。\n", "\n", "## 执行方式\n", "\n", "这里演示在Graph模式和PyNative模式下，单算子、普通函数、模型的执行方式。\n", "\n", "> 在本案例的实际执行中，采取了MindSpore的默认方式`GRAPH_MODE`，用户也可以将其变更为`PYNATIVE_MODE`进行尝试。" ] }, { "cell_type": "code", "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "import mindspore.nn as nn\n", "from mindspore import context, Tensor, ParameterTuple, ms_function\n", "import mindspore.ops as ops\n", "from mindspore import dtype as mstype\n", "from mindspore.common.initializer import Normal\n", "from mindspore.nn import Dense, WithLossCell, SoftmaxCrossEntropyWithLogits, Momentum\n", "\n", "# 设定为Graph模式，也可替换为PYNATIVE_MODE\n", "context.set_context(mode=context.GRAPH_MODE, device_target=\"Ascend\")" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "### 执行单算子" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[[[-0.02593261 0.01615404 0.01615404 0.01615404 0.01196378]\n", " [-0.01535788 0.05602208 0.05602208 0.05602208 0.04094065]\n", " [-0.01535788 0.05602208 0.05602208 0.05602208 0.04094065]\n", " [-0.01535788 0.05602208 0.05602208 0.05602208 0.04094065]\n", " [-0.01409336 0.04544117 0.04544117 0.04544117 0.0373004 ]]\n", "\n", " [[ 0.03874376 0.02201786 0.02201786 0.02201786 0.02687691]\n", " [ 0.05751193 0.02690699 0.02690699 0.02690699 0.03515062]\n", " [ 0.05751193 0.02690699 0.02690699 0.02690699 0.03515062]\n", " [ 0.05751193 0.02690699 0.02690699 0.02690699 0.03515062]\n", " [ 0.02599058 0.01130002 0.01130002 0.01130002 0.02304572]]\n", "\n", " [[-0.00022919 0.02640852 0.02640852 0.02640852 0.04932421]\n", " [ 0.01657246 0.0705748 0.0705748 0.0705748 0.0874946 ]\n", " [ 0.01657246 0.0705748 0.0705748 0.0705748 0.0874946 ]\n", " [ 0.01657246 0.0705748 0.0705748 0.0705748 0.0874946 ]\n", " [ 0.03821789 0.09614976 0.09614976 0.09614976 0.10491695]]\n", "\n", " [[ 0.0190958 0.02602289 0.02602289 0.02602289 0.01660084]\n", " [ 0.03556763 0.06862713 0.06862713 0.06862713 0.02653556]\n", " [ 0.03556763 0.06862713 0.06862713 0.06862713 0.02653556]\n", " [ 0.03556763 0.06862713 0.06862713 0.06862713 0.02653556]\n", " [ 0.00727296 0.04514674 0.04514674 0.04514674 0.01423099]]]]\n" ] } ], "source": [ "# 打印Cond2d算子的输出\n", "conv = nn.Conv2d(3, 4, 3, bias_init='zeros')\n", "input_data = Tensor(np.ones([1, 3, 5, 5]).astype(np.float32))\n", "output = conv(input_data)\n", "print(output.asnumpy())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 执行普通函数\n", "\n", "将若干算子组合成一个函数，然后直接通过函数调用的方式执行这些算子，并打印相关结果，如下例所示。\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[3. 3. 3.]\n", " [3. 3. 3.]\n", " [3. 3. 3.]]\n" ] } ], "source": [ "import numpy as np\n", "from mindspore import context, Tensor\n", "import mindspore.ops as ops\n", "\n", "context.set_context(mode=context.PYNATIVE_MODE, device_target=\"CPU\")\n", "\n", "def add_func(x, y):\n", " z = ops.add(x, y)\n", " z = ops.add(z, x)\n", " return z\n", "\n", "x = Tensor(np.ones([3, 3], dtype=np.float32))\n", "y = Tensor(np.ones([3, 3], dtype=np.float32))\n", "output = add_func(x, y)\n", "print(output.asnumpy())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 执行网络" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ 4. 10. 18.]\n" ] } ], "source": [ "import numpy as np\n", "import mindspore.nn as nn\n", "import mindspore.ops as ops\n", "from mindspore import context, Tensor\n", "\n", "class Net(nn.Cell):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", " self.mul = ops.Mul()\n", "\n", " def construct(self, x, y):\n", " return self.mul(x, y)\n", "\n", "x = Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n", "y = Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32))\n", "\n", "net = Net()\n", "print(net(x, y))" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "## PyNative模式说明\n", "\n", "### 性能优化\n", "\n", "正如文章开头所说，Graph模式适合高性能的场景，但PyNative模式中也提供了性能优化的手段。MindSpore提供了Staging功能，该功能可以在PyNative模式下将Python函数或者Python类的方法编译成计算图，通过图优化等技术提高运行速度，是一种混合运行机制。Staging功能的使用通过`ms_function`装饰器达成，该装饰器会将模块编译成计算图，在给定输入之后，以图的形式下发执行。如下例所示：" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[2. 2. 2. 2.]\n", " [2. 2. 2. 2.]\n", " [2. 2. 2. 2.]\n", " [2. 2. 2. 2.]]\n" ] } ], "source": [ "# 导入ms_function\n", "from mindspore import ms_function\n", "\n", "# 仍设定为PyNative模式\n", "context.set_context(mode=context.PYNATIVE_MODE, device_target=\"Ascend\")\n", "\n", "add = ops.Add()\n", "\n", "# 使用装饰器编译计算图\n", "@ms_function\n", "def add_fn(x, y):\n", " res = add(x, y)\n", " return res\n", "\n", "x = Tensor(np.ones([4, 4]).astype(np.float32))\n", "y = Tensor(np.ones([4, 4]).astype(np.float32))\n", "z = add_fn(x, y)\n", "print(z.asnumpy())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在加装了`ms_function`装饰器的函数中，如果包含不需要进行参数训练的算子（如`pooling`、`add`等算子），则这些算子可以在被装饰的函数中直接调用，如上例所示。如果被装饰的函数中包含了需要进行参数训练的算子（如`Convolution`、`BatchNorm`等算子），则这些算子必须在被装饰等函数之外完成实例化操作。" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[[[ 0.00994964 0.01850731 -0.05146599]\n", " [ 0.02427048 -0.09082688 -0.00945184]\n", " [ 0.02710651 -0.07322617 0.02594434]]\n", "\n", " [[ 0.00056772 -0.05043615 -0.03873939]\n", " [-0.00445028 0.03694705 -0.03555503]\n", " [ 0.07329068 -0.02026664 0.01922888]]\n", "\n", " [[ 0.02257145 -0.04093865 -0.00493869]\n", " [ 0.01740007 0.02478302 0.02072578]\n", " [ 0.05831327 -0.03933404 0.01767443]]\n", "\n", " [[-0.03954437 0.02160874 -0.00700614]\n", " [ 0.03856367 -0.04015685 0.02508826]\n", " [-0.0229507 -0.03803677 0.02813173]]]\n", "\n", "\n", " [[[ 0.01678797 -0.02227589 -0.04470547]\n", " [-0.05720481 -0.15464461 0.00911596]\n", " [ 0.02566019 -0.04340314 0.03164666]]\n", "\n", " [[ 0.03300299 -0.05849815 0.05841954]\n", " [-0.11595733 -0.01524522 0.02947116]\n", " [ 0.05930116 0.00831041 -0.0466827 ]]\n", "\n", " [[-0.0797728 0.02910854 0.00766015]\n", " [-0.01380327 -0.03338642 0.02625138]\n", " [ 0.02279372 -0.00952736 0.02026749]]\n", "\n", " [[ 0.04039776 -0.05340278 -0.0083563 ]\n", " [ 0.04991922 -0.05205034 -0.0058607 ]\n", " [ 0.00686666 0.00064385 0.00301326]]]]\n" ] } ], "source": [ "# Conv2d实例化操作\n", "conv_obj = nn.Conv2d(in_channels=3, out_channels=4, kernel_size=3, stride=2, padding=0)\n", "conv_obj.init_parameters_data()\n", "\n", "@ms_function\n", "def conv_fn(x):\n", " res = conv_obj(x)\n", " return res\n", "\n", "input_data = np.random.randn(2, 3, 6, 6).astype(np.float32)\n", "z = conv_fn(Tensor(input_data))\n", "print(z.asnumpy())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 梯度求取\n", "\n", "PyNative模式中支持单独的梯度求取操作，下面演示如何利用这一特性调试网络模型。具体操作可通过`GradOperation`求该函数或者网络所有的输入梯度。需要注意，输入类型仅支持Tensor。\n", "\n", "构建网络如下。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "class LeNet5(nn.Cell):\n", " \"\"\"\n", " Lenet网络结构\n", " \"\"\"\n", " def __init__(self, num_class=10, num_channel=1):\n", " super(LeNet5, self).__init__()\n", " # 定义所需要的运算\n", " self.conv1 = nn.Conv2d(num_channel, 6, 5, pad_mode='valid')\n", " self.conv2 = nn.Conv2d(6, 16, 5, pad_mode='valid')\n", " self.fc1 = nn.Dense(16 * 5 * 5, 120, weight_init=Normal(0.02))\n", " self.fc2 = nn.Dense(120, 84, weight_init=Normal(0.02))\n", " self.fc3 = nn.Dense(84, num_class, weight_init=Normal(0.02))\n", " self.relu = nn.ReLU()\n", " self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)\n", " self.flatten = nn.Flatten()\n", "\n", " def construct(self, x):\n", " # 使用定义好的运算构建前向网络\n", " x = self.conv1(x)\n", " x = self.relu(x)\n", " x = self.max_pool2d(x)\n", " x = self.conv2(x)\n", " x = self.relu(x)\n", " x = self.max_pool2d(x)\n", " x = self.flatten(x)\n", " x = self.fc1(x)\n", " x = self.relu(x)\n", " x = self.fc2(x)\n", " x = self.relu(x)\n", " x = self.fc3(x)\n", " return x\n", "\n", "# 实例化网络\n", "net = LeNet5()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "如上文所说，利用`GradOperation`求函数输入梯度。" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "class GradWrap(nn.Cell):\n", " \"\"\"求函数输入梯度\"\"\"\n", " def __init__(self, network):\n", " super(GradWrap, self).__init__(auto_prefix=False)\n", " self.network = network\n", " # 用Tuple的形式包装weight\n", " self.weights = ParameterTuple(filter(lambda x: x.requires_grad, network.get_parameters()))\n", "\n", " def construct(self, x, label):\n", " weights = self.weights\n", " # 返回值为梯度\n", " return ops.GradOperation(get_by_list=True)(self.network, weights)(x, label)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在PyNative模式中进行网络训练。" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2.3025854\n" ] } ], "source": [ "# 设定优化器、损失函数\n", "optimizer = Momentum(filter(lambda x: x.requires_grad, net.get_parameters()), 0.1, 0.9)\n", "criterion = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean')\n", "\n", "# 通过WithLossCell获取Loss值\n", "net_with_criterion = WithLossCell(net, criterion)\n", "\n", "# 调用GradWrap\n", "train_network = GradWrap(net_with_criterion)\n", "train_network.set_train()\n", "\n", "# 产生输入数据\n", "input_data = Tensor(np.ones([32, 1, 32, 32]).astype(np.float32) * 0.01)\n", "label = Tensor(np.ones([32]).astype(np.int32))\n", "output = net(Tensor(input_data))\n", "\n", "# 利用前向网络计算loss\n", "loss_output = criterion(output, label)\n", "# 求得梯度\n", "grads = train_network(input_data, label)\n", "success = optimizer(grads)\n", "#输出loss\n", "loss = loss_output.asnumpy()\n", "print(loss)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.5" } }, "nbformat": 4, "nbformat_minor": 4 }