{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 动静态图\n",
    "\n",
    "[![下载Notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/resource/_static/logo_notebook.png)](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r1.8/tutorials/zh_cn/advanced/pynative_graph/mindspore_mode.ipynb) \n",
    "[![下载样例代码](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/resource/_static/logo_download_code.png)](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r1.8/tutorials/zh_cn/advanced/pynative_graph/mindspore_mode.py) \n",
    "[![查看源文件](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r1.8/tutorials/source_zh_cn/advanced/pynative_graph/mode.ipynb)\n",
    "\n",
    "目前主流的深度学习框架有静态图(Graph)和动态图(PyNative)两种执行模式。\n",
    "\n",
    "- 静态图模式下，程序在编译执行时，首先生成神经网络的图结构，然后再执行图中涉及的计算操作。因此，在静态图模式下，编译器可以通过使用图优化等技术来获得更好的执行性能，有助于规模部署和跨平台运行。\n",
    "\n",
    "- 动态图模式下，程序按照代码的编写顺序逐行执行，在执行正向过程中根据反向传播的原理，动态生成反向执行图。这种模式下，编译器将神经网络中的各个算子逐一下发到设备进行计算操作，方便用户编写和调试神经网络模型。\n",
    "\n",
    "## 动静态图介绍\n",
    "\n",
    "MindSpore提供了静态图和动态图统一的编码方式，大大增加了静态图和动态图的可兼容性，用户无需开发多套代码，仅变更一行代码便可切换静态图/动态图模式。静态图模式是MindSpore的默认模式，而动态图模式用于调试等用途。\n",
    "\n",
    "> 当运行模式从动态图切换到静态图时，请留意[静态图语法支持](https://www.mindspore.cn/docs/zh-CN/r1.8/note/static_graph_syntax_support.html)。\n",
    "\n",
    "### 模式选择\n",
    "\n",
    "通过配置context参数可以控制程序运行的模式，动态图和静态图两种模式的区别主要有：\n",
    "\n",
    "- **适用场景**：静态图需要一开始就构建好网络结构，然后框架做整图优化和执行，比较适合网络固定没有变化，且需要高性能的场景。动态图逐行执行算子，支持执行单算子、普通函数和网络，以及单独求梯度的操作。\n",
    "\n",
    "- **网络执行**：静态图模式和动态图模式在执行相同的网络和算子时，精度效果一致。由于静态图模式运用了图优化、计算图整图下沉等技术，静态图模式执行网络的性能和效率更高，动态图模式更便于调试调优。\n",
    "\n",
    "- **代码调试**：在脚本开发和网络流程调试中，推荐使用动态图模式进行调试。在动态图模式下，可以方便地设置断点，获取网络执行的中间结果，也可以通过pdb的方式对网络进行调试。而静态图模式无法设置断点，只能先指定算子进行打印，然后在网络执行完成后查看输出结果。\n",
    "\n",
    "### 模式切换\n",
    "\n",
    "模式切换时，需要设置context中的运行模式。首先定义网络模型`MyNet`和后续代码片段用到的数据，用于后续的动静态图模式的切换和展示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import mindspore.nn as nn\n",
    "import mindspore.ops as ops\n",
    "import mindspore as ms\n",
    "\n",
    "class MyNet(nn.Cell):\n",
    "    \"\"\"自定义网络，实现两个张量的加法\"\"\"\n",
    "    def __init__(self):\n",
    "        super(MyNet, self).__init__()\n",
    "        self.add = ops.Add()\n",
    "\n",
    "    def construct(self, x, y):\n",
    "        return self.add(x, y)\n",
    "\n",
    "x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n",
    "y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设置运行模式为静态图模式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5. 7. 9.]\n"
     ]
    }
   ],
   "source": [
    "import mindspore as ms\n",
    "\n",
    "ms.set_context(mode=ms.GRAPH_MODE)\n",
    "\n",
    "net = MyNet()\n",
    "print(net(x, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MindSpore处于静态图模式时，可以通过`mode=ms.PYNATIVE_MODE`切换为动态图模式；同样，MindSpore处于动态图模式时，可以通过`mode=ms.GRAPH_MODE`切换为静态图模式，请留意[静态图语法支持](https://www.mindspore.cn/docs/zh-CN/r1.8/note/static_graph_syntax_support.html)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5. 7. 9.]\n"
     ]
    }
   ],
   "source": [
    "ms.set_context(mode=ms.PYNATIVE_MODE)\n",
    "\n",
    "net = MyNet()\n",
    "print(net(x, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 静态图\n",
    "\n",
    "在MindSpore中，静态图模式又被称为Graph模式，比较适合网络固定且需要高性能的场景，可以通过`set_context`接口中，参数`mode`入参为`GRAPH_MODE`来设置成静态图模式。\n",
    "\n",
    "在静态图模式下，基于图优化、计算图整图下沉等技术，编译器可以针对图进行全局的优化，因此在静态图模式下执行时可以获得较好的性能。但是，执行图是从源码转换而来，因此在静态图模式下不是所有的Python语法都能支持，会有一些特殊的约束，其支持情况的详细信息可参考[静态图语法支持](https://www.mindspore.cn/docs/zh-CN/r1.8/note/static_graph_syntax_support.html)。\n",
    "\n",
    "### 静态图模式执行原理\n",
    "\n",
    "在静态图模式下，MindSpore通过源码转换的方式，将Python的源码转换成中间表达形式，也就是IR（Intermediate Representation），并在此基础上对IR图进行优化，最终在硬件设备上执行优化后的图。\n",
    "\n",
    "MindSpore使用的是一种基于图表示的函数式IR，称为MindIR。静态图模式就是基于MindIR进行编译优化，使用静态图模式时，需要使用[nn.Cell](https://mindspore.cn/docs/zh-CN/r1.8/api_python/nn/mindspore.nn.Cell.html#mindspore.nn.Cell)类并且在`construct`函数中编写执行代码。\n",
    "\n",
    "### 静态图模式代码示例\n",
    "\n",
    "静态图模式的代码用例如下所示，神经网络模型实现 $f(x, y)=x*y$ 的计算操作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-04T10:51:23.282871Z",
     "start_time": "2022-01-04T10:51:21.743620Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 4. 10. 18.]\n"
     ]
    }
   ],
   "source": [
    "# 设置运行模式为静态图模式\n",
    "ms.set_context(mode=ms.GRAPH_MODE)\n",
    "\n",
    "class Net(nn.Cell):\n",
    "    \"\"\"自定义网络，实现两个张量的乘法\"\"\"\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.mul = ops.Mul()\n",
    "\n",
    "    def construct(self, x, y):\n",
    "        \"\"\"定义执行代码\"\"\"\n",
    "        return self.mul(x, y)\n",
    "\n",
    "x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n",
    "y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32))\n",
    "\n",
    "net = Net()\n",
    "\n",
    "print(net(x, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 静态图模式下的控制流\n",
    "\n",
    "请参考[流程控制](https://www.mindspore.cn/tutorials/zh-CN/r1.8/advanced/network/control_flow.html)，阅读更多静态图模式下的控制流。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 动态图\n",
    "\n",
    "在MindSpore中，动态图模式又被称为PyNative模式，可以通过`set_context`接口中，参数`mode`入参为`PYNATIVE_MODE`来设置成动态图模式。\n",
    "\n",
    "在脚本开发和网络流程调试中，推荐使用动态图模式进行调试，其支持执行单算子、普通函数和网络、以及单独求梯度的操作。\n",
    "\n",
    "### 动态图模式执行原理\n",
    "\n",
    "在动态图模式下，用户可以使用完整的Python API，此外针对使用MindSpore提供的API时，框架会根据用户选择的不同硬件平台（Ascend/GPU/CPU）或环境信息，将算子API的操作在对应的硬件平台上执行，并返回相应的结果。\n",
    "\n",
    "框架整体的执行过程如下：\n",
    "\n",
    "![process](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/tutorials/source_zh_cn/advanced/pynative_graph/images/framework2.png)\n",
    "\n",
    "通过前端的Python API，调用到框架层，最终到相应的硬件设备上进行计算。\n",
    "\n",
    "下面我们通过`ops.mul`算子，直接代替静态图模式下需要定义网络模型，实现 $f(x, y)=x*y$ 的计算操作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-04T10:51:23.292278Z",
     "start_time": "2022-01-04T10:51:23.284465Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 4. 10. 18.]\n"
     ]
    }
   ],
   "source": [
    "# 设置运行模式为动态图模式\n",
    "ms.set_context(mode=ms.PYNATIVE_MODE)\n",
    "\n",
    "x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n",
    "y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32))\n",
    "\n",
    "output = ops.mul(x, y)\n",
    "\n",
    "print(output.asnumpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在上面的示例代码中，当调用到API接口`ops.mul(x, y)`时，会将MindSpore表达层Python API接口的调用，通过[Pybind11](https://pybind11.readthedocs.io/en/stable/basics.html)调用到MindSpore框架的C++层，转换成C++的接口调用。接着框架会根据MindSpore的安装环境信息，自动选择对应的硬件设备，在该硬件设备上执行add操作。\n",
    "\n",
    "从上述原理可以看到，PyNative模式下，Python脚本代码会根据Python的语法进行执行，而执行过程中涉及到MindSpore表达层的Python API，会根据用户的设置在不同的硬件上执行，从而进行性能加速。\n",
    "\n",
    "因此，在动态图模式下，用户可以随意使用Python的语法以及调试方法。\n",
    "\n",
    "### 动态图模式自动微分原理\n",
    "\n",
    "在动态图下，执行正向过程完全是按照Python的语法执行的，而反向传播过程是基于Tensor实现的。\n",
    "\n",
    "因此，我们在执行正向过程中，将所有应用于Tensor的操作记录下来，并针对每个计算操作求取其反向，然后将所有反向过程串联起来形成整体反向传播图，最终将反向图在设备上执行并计算出梯度。\n",
    "\n",
    "下面通过一段简单的示例代码说明动态图模式自动微分原理。对矩阵x乘上固定参数z，然后与y进行矩阵乘法：\n",
    "\n",
    "$$f(x, y)=(x * z) * y \\tag{1}$$\n",
    "\n",
    "代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-04T10:51:23.439867Z",
     "start_time": "2022-01-04T10:51:23.293334Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[4.5099998 2.7       3.6000001]\n",
      " [4.5099998 2.7       3.6000001]]\n"
     ]
    }
   ],
   "source": [
    "# 设置运行模式为动态图模式\n",
    "ms.set_context(mode=ms.PYNATIVE_MODE)\n",
    "\n",
    "class Net(nn.Cell):\n",
    "    \"\"\"自定义网络\"\"\"\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.matmul = ops.MatMul()\n",
    "        self.z = ms.Parameter(ms.Tensor(np.array([1.0], np.float32)), name='z')\n",
    "\n",
    "    def construct(self, x, y):\n",
    "        x = x * self.z\n",
    "        x = self.matmul(x, y)\n",
    "        return x\n",
    "\n",
    "class GradNetWrtX(nn.Cell):\n",
    "    \"\"\"定义对x的求导\"\"\"\n",
    "    def __init__(self, net):\n",
    "        super(GradNetWrtX, self).__init__()\n",
    "\n",
    "        self.net = net\n",
    "        self.grad_op = ops.GradOperation()\n",
    "\n",
    "    def construct(self, x, y):\n",
    "        gradient_function = self.grad_op(self.net)\n",
    "        return gradient_function(x, y)\n",
    "\n",
    "x = ms.Tensor([[0.8, 0.6, 0.2], [1.8, 1.3, 1.1]], dtype=ms.float32)\n",
    "y = ms.Tensor([[0.11, 3.3, 1.1], [1.1, 0.2, 1.4], [1.1, 2.2, 0.3]], dtype=ms.float32)\n",
    "\n",
    "output = GradNetWrtX(Net())(x, y)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 由于不同计算平台的精度可能存在差异，因此上面的代码在不同平台上的执行结果会存在微小的差别。求导公式的推导和上述打印结果的解释，可参考[自动求导](https://www.mindspore.cn/tutorials/zh-CN/r1.8/advanced/network/derivation.html#一阶求导)章节。\n",
    "\n",
    "![forward](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/tutorials/source_zh_cn/advanced/pynative_graph/images/forward_backward.png)\n",
    "\n",
    "根据上述动态图模式下构图原理可以看到，在正向传播过程中，MindSpore记录了Mul的计算过程，根据Mul对应的反向bprop的定义，得到了反向的MulGrad算子。\n",
    "\n",
    "根据Mul算子的bprop定义，如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mindspore.ops._grad.grad_base import bprop_getters\n",
    "\n",
    "@bprop_getters.register(ops.Mul)\n",
    "def get_bprop_mul(self):\n",
    "    \"\"\"Grad definition for `Mul` operation.\"\"\"\n",
    "    mul_func = P.Mul()\n",
    "\n",
    "    def bprop(x, y, out, dout):\n",
    "        bc_dx = mul_func(y, dout)\n",
    "        bc_dy = mul_func(x, dout)\n",
    "        return binop_grad_common(x, y, bc_dx, bc_dy)\n",
    "\n",
    "    return bprop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到对Mul的输入求反向，需要两个输入和输出的反向传播梯度值，此时根据实际的输入值，可以将z连接到MulGrad。以此类推，对下一个算子Matmul，相应的得到MatmulGrad信息，再根据bprop的输入输出，将上下文梯度传播连接起来。\n",
    "\n",
    "同理对于输入y求导，可以使用同样的过程进行推导。\n",
    "\n",
    "### 动态图模式下的控制流\n",
    "\n",
    "在MindSpore中，针对控制流语法并没有做特殊处理，直接按照Python的语法展开执行，进而对展开的执行算子进行自动微分操作。\n",
    "\n",
    "例如，对于for循环，在动态图下会首先执行Python的源码，然后根据具体的循环次数，不断的执行for循环中的语句，并对其算子进行自动微分操作。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4. 5. 6.]\n"
     ]
    }
   ],
   "source": [
    "# 设置运行模式为动态图模式\n",
    "ms.set_context(mode=ms.PYNATIVE_MODE)\n",
    "\n",
    "class Net(nn.Cell):\n",
    "    \"\"\"自定义网络\"\"\"\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.matmul = ops.MatMul()\n",
    "        self.z = ms.Parameter(ms.Tensor(np.array([1.0], np.float32)), name='z')\n",
    "\n",
    "    def construct(self, x):\n",
    "        for _ in range(3):\n",
    "            x = x + self.z\n",
    "        return x\n",
    "\n",
    "x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n",
    "net = Net()\n",
    "output = net(x)\n",
    "\n",
    "print(output)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MindSpore",
   "language": "python",
   "name": "mindspore"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}