{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 动静态图结合\n", "\n", "[![下载Notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0/resource/_static/logo_notebook.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r2.0/zh_cn/design/mindspore_dynamic_graph_and_static_graph.ipynb) [![下载样例代码](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0/resource/_static/logo_download_code.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r2.0/zh_cn/design/mindspore_dynamic_graph_and_static_graph.py) [![查看源文件](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r2.0/docs/mindspore/source_zh_cn/design/dynamic_graph_and_static_graph.ipynb)\n", "\n", "## 静态图和动态图的概念\n", "\n", "目前主流的深度学习框架的执行模式有两种，分别为静态图模式和动态图模式。\n", "\n", "静态图模式下，程序在编译执行时先生成神经网络的图结构，然后再执行图中涉及的计算操作。因此，在静态图模式下，编译器利用图优化等技术对执行图进行更大程度的优化，从而获得更好的执行性能，有助于规模部署和跨平台运行。\n", "\n", "动态图模式下，程序按照代码的编写顺序执行，在执行正向过程中根据反向传播的原理，动态生成反向执行图。这种模式下，编译器将神经网络中的各个算子逐一下发执行，方便用户编写和调试神经网络模型。\n", "\n", "## MindSpore静态图\n", "\n", "在MindSpore中，静态图模式又被称为Graph模式，可以通过`set_context(mode=GRAPH_MODE)`来设置成静态图模式。静态图模式比较适合网络固定且需要高性能的场景。在静态图模式下，基于图优化、计算图整图下沉等技术，编译器可以针对图进行全局的优化，因此在静态图下能获得较好的性能，但是执行图是从源码转换而来，因此在静态图下不是所有的Python语法都能支持。\n", "\n", "### Graph模式执行原理\n", "\n", "在Graph模式下，MindSpore通过源码转换的方式，将Python的源码转换成IR，再在此基础上进行相关的图优化，最终在硬件设备上执行优化后的图。MindSpore使用的是一种基于图表示的函数式IR，即MindIR，采用了接近于ANF函数式的语义。Graph模式是基于MindIR进行编译优化的，使用Graph模式时，需要使用`nn.Cell`类并且在`construct`函数中编写执行代码，或者调用`@jit`装饰器。\n", "\n", "Graph模式的代码用例如下所示：" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T10:51:23.282871Z", "start_time": "2022-01-04T10:51:21.743620Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ 4. 10. 18.]\n" ] } ], "source": [ "import numpy as np\n", "import mindspore.nn as nn\n", "import mindspore.ops as ops\n", "import mindspore as ms\n", "\n", "ms.set_context(mode=ms.GRAPH_MODE, device_target=\"CPU\")\n", "\n", "class Net(nn.Cell):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", " self.mul = ops.Mul()\n", "\n", " def construct(self, x, y):\n", " return self.mul(x, y)\n", "\n", "x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n", "y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32))\n", "\n", "net = Net()\n", "print(net(x, y))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Graph模式自动微分原理\n", "\n", "在MindSpore中，Graph模式下的自动微分原理可以参考[自动微分](https://www.mindspore.cn/tutorials/zh-CN/r2.0/beginner/autograd.html)。\n", "\n", "## MindSpore动态图\n", "\n", "在MindSpore中，动态图模式又被称为PyNative模式，可以通过`set_context(mode=PYNATIVE_MODE)`来设置成动态图模式。在脚本开发和网络流程调试中，推荐使用动态图模式进行调试，支持执行单算子、普通函数和网络、以及单独求梯度的操作。\n", "\n", "### PyNative模式执行原理\n", "\n", "在PyNative模式下，用户可以使用完整的Python API，此外针对使用MindSpore提供的API时，框架会根据用户选择的硬件平台（Ascend，GPU，CPU），将算子API的操作在对应的硬件平台上执行，并返回相应的结果。框架整体的执行过程如下：\n", "\n", "![process](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0/docs/mindspore/source_zh_cn/design/images/framework.png)\n", "\n", "通过前端的Python API，调用到框架层，最终到相应的硬件设备上进行计算。例如：完成一个加法" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T10:51:23.292278Z", "start_time": "2022-01-04T10:51:23.284465Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[[[2. 2. 2. 2.]\n", " [2. 2. 2. 2.]\n", " [2. 2. 2. 2.]]\n", "\n", " [[2. 2. 2. 2.]\n", " [2. 2. 2. 2.]\n", " [2. 2. 2. 2.]]\n", "\n", " [[2. 2. 2. 2.]\n", " [2. 2. 2. 2.]\n", " [2. 2. 2. 2.]]]]\n" ] } ], "source": [ "import numpy as np\n", "import mindspore.nn as nn\n", "import mindspore as ms\n", "import mindspore.ops as ops\n", "\n", "ms.set_context(mode=ms.PYNATIVE_MODE, device_target=\"CPU\")\n", "x = ms.Tensor(np.ones([1, 3, 3, 4]).astype(np.float32))\n", "y = ms.Tensor(np.ones([1, 3, 3, 4]).astype(np.float32))\n", "output = ops.add(x, y)\n", "print(output.asnumpy())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "此例中，当调用到Python接口ops.add(x, y)时，会将Python的接口调用通过Pybind11调用到框架的C++层，转换成C++的调用，接着框架会根据用户设置的device_target选择对应的硬件设备，在该硬件设备上执行add这个操作。\n", "\n", "从上述原理可以看到，在PyNative模式下，Python脚本代码会根据Python的语法进行执行，而执行过程中涉及到MindSpore的API，会根据用户设置在不同的硬件上进行执行，从而进行加速。因此，在PyNative模式下，用户可以随意使用Python的语法以及调试方法。例如可以使用常见的PyCharm、VS Code等IDE进行代码的调试。\n", "\n", "### PyNative模式自动微分原理\n", "\n", "在前面的介绍中，我们可以看出，在PyNative下执行正向过程完全是按照Python的语法进行执行。在PyNative下是基于Tensor进行实现反向传播的，我们在执行正向过程中，将所有应用于Tensor的操作记录下来，并针对每个操作求取其反向，并将所有反向过程串联起来形成整体反向传播图（简称反向图）。最终，将反向图在设备上进行执行计算出梯度。\n", "\n", "反向构图过程示例，如下代码，对矩阵x乘上固定参数z，然后与y进行矩阵乘法，最终对x进行求导。" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T10:51:23.439867Z", "start_time": "2022-01-04T10:51:23.293334Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[9.02 5.4 7.2000003]\n", " [9.02 5.4 7.2000003]]\n" ] } ], "source": [ "import numpy as np\n", "import mindspore.nn as nn\n", "import mindspore.ops as ops\n", "import mindspore as ms\n", "\n", "ms.set_context(mode=ms.PYNATIVE_MODE, device_target=\"CPU\")\n", "\n", "class Net(nn.Cell):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", " self.matmul = ops.MatMul()\n", " self.z = ms.Parameter(ms.Tensor(np.array([2.0], np.float32)), name='z')\n", "\n", " def construct(self, x, y):\n", " x = x * self.z\n", " out = self.matmul(x, y)\n", " return out\n", "\n", "class GradNetWrtX(nn.Cell):\n", " def __init__(self, net):\n", " super(GradNetWrtX, self).__init__()\n", " self.net = net\n", "\n", " def construct(self, x, y):\n", " gradient_function = ms.grad(self.net)\n", " return gradient_function(x, y)\n", "\n", "x = ms.Tensor([[0.8, 0.6, 0.2], [1.8, 1.3, 1.1]], dtype=ms.float32)\n", "y = ms.Tensor([[0.11, 3.3, 1.1], [1.1, 0.2, 1.4], [1.1, 2.2, 0.3]], dtype=ms.float32)\n", "output = GradNetWrtX(Net())(x, y)\n", "print(output)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![forward](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0/docs/mindspore/source_zh_cn/design/images/forward.png) ![backward](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0/docs/mindspore/source_zh_cn/design/images/backward.png)\n", "\n", "根据上述PyNative下构图原理，我们可以看到，在正向传播过程中，我们记录了Mul的计算过程，根据Mul对应的反向bprop的定义，得到了反向的MulGrad算子，根据Mul算子的bprop定义，如下：" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T10:51:23.445921Z", "start_time": "2022-01-04T10:51:23.441876Z" } }, "outputs": [], "source": [ "from mindspore.ops._grad.grad_base import bprop_getters\n", "\n", "@bprop_getters.register(ops.Mul)\n", "def get_bprop_mul(self):\n", " \"\"\"Grad definition for `Mul` operation.\"\"\"\n", " mul_func = P.Mul()\n", "\n", " def bprop(x, y, out, dout):\n", " bc_dx = mul_func(y, dout)\n", " bc_dy = mul_func(x, dout)\n", " return binop_grad_common(x, y, bc_dx, bc_dy)\n", "\n", " return bprop" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "可以看到对Mul的输入求反向，需要两个输入和输出的反向传播梯度值，此时根据实际的输入值，可以将z连接到MulGrad。以此类推，对下一个算子Matmul，相应的得到MatmulGrad信息，再根据bprop的输入输出，将上下文梯度传播连接起来。\n", "\n", "同理对于输入y求导，可以使用同样的过程进行推导。\n", "\n", "### PyNative模式下的控制流\n", "\n", "在PyNative模式下，脚本按照Python的语法执行，因此在MindSpore中，针对控制流语法并没有做特殊处理，直接按照Python的语法直接展开执行，进而对展开的执行算子进行自动微分的操作。例如，对于for循环，在PyNative下会根据具体的循环次数，不断的执行for循环中的语句，并对其算子进行自动微分的操作。\n", "\n", "## 动静统一\n", "\n", "### 概述\n", "\n", "当前在业界支持动态图和静态图两种模式，动态图通过解释执行，具有动态语法亲和性，表达灵活；静态图使用jit编译优化执行，偏静态语法，在语法上有较多限制。动态图和静态图的编译流程不一致，语法约束不一致。MindSpore针对动态图和静态图模式，首先统一API表达，在两种模式下使用相同的API；其次统一动态图和静态图的底层微分机制。\n", "\n", "![dynamic](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0/docs/mindspore/source_zh_cn/design/images/dynamic.png)\n", "\n", "### 动态图和静态图互相转换\n", "\n", "在MindSpore中，我们可以通过控制模式输入参数来切换执行使用动态图还是静态图。例如：" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T10:51:23.461198Z", "start_time": "2022-01-04T10:51:23.447508Z" } }, "outputs": [], "source": [ "ms.set_context(mode=ms.PYNATIVE_MODE)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "由于在静态图下，对于Python语法有所限制，因此从动态图切换成静态图时，需要符合静态图的语法限制，才能正确使用静态图来进行执行。更多静态图的语法限制可以参考[静态图语法限制](https://www.mindspore.cn/docs/zh-CN/r2.0/note/static_graph_syntax_support.html)。\n", "\n", "### 动静结合\n", "\n", "MindSpore支持在动态图下使用静态编译的方式来进行混合执行，通过使用jit修饰需要用静态图来执行的函数对象，即可实现动态图和静态图的混合执行，更多jit的使用可参考[jit文档](https://www.mindspore.cn/tutorials/zh-CN/r2.0/advanced/compute_graph.html#即时编译)。\n", "\n", "例如：" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T10:51:23.514919Z", "start_time": "2022-01-04T10:51:23.462207Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[[[15.99984]]\n", "\n", " [[15.99984]]]]\n" ] } ], "source": [ "import numpy as np\n", "import mindspore as ms\n", "import mindspore.nn as nn\n", "\n", "class AddMulMul(nn.Cell):\n", " def __init__(self):\n", " super(AddMulMul, self).__init__()\n", " self.param = ms.Parameter(ms.Tensor(0.5, ms.float32))\n", "\n", " @ms.jit\n", " def construct(self, x):\n", " x = x + x\n", " x = x * self.param\n", " x = x * x\n", " return x\n", "\n", "class CellCallSingleCell(nn.Cell):\n", " def __init__(self):\n", " super(CellCallSingleCell, self).__init__()\n", " self.conv = nn.Conv2d(1, 2, kernel_size=2, stride=1, padding=0, weight_init=\"ones\", pad_mode=\"valid\")\n", " self.bn = nn.BatchNorm2d(2, momentum=0.99, eps=0.00001, gamma_init=\"ones\")\n", " self.relu = nn.ReLU()\n", " self.add_mul_mul = AddMulMul()\n", "\n", " def construct(self, x):\n", " x = self.conv(x)\n", " x = self.bn(x)\n", " x = self.add_mul_mul(x)\n", " x = self.relu(x)\n", " return x\n", "\n", "ms.set_context(mode=ms.PYNATIVE_MODE, device_target=\"CPU\")\n", "inputs = ms.Tensor(np.ones([1, 1, 2, 2]).astype(np.float32))\n", "net = CellCallSingleCell()\n", "out = net(inputs)\n", "print(out)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### JIT Fallback\n", "\n", "在MindSpore静态图模式下，用户编写程序时需要遵循MindSpore[静态图语法支持](https://www.mindspore.cn/docs/zh-CN/r2.0/note/static_graph_syntax_support.html)，语法使用存在约束限制。而在动态图模式下，Python脚本代码会根据Python语法进行执行，用户可以使用任意Python语法。可以看出，静态图和动态图的语法约束限制是不同的。\n", "\n", "JIT Fallback是从静态图的角度出发考虑静态图和动态图的统一。通过JIT Fallback特性，静态图可以支持尽量多的动态图语法，使得静态图提供接近动态图的语法使用体验，从而实现动静统一。为了便于用户选择是否使用JIT Fallback特性的能力，提供了开关`MS_DEV_ENABLE_FALLBACK`，当前默认已经打开。如果需要关闭，可以使用命令：`export MS_DEV_ENABLE_FALLBACK=0`。\n", "\n", "下面主要介绍JIT Fallback的支持范围和使用须知，以便您可以更有效地使用JIT Fallback功能。\n", "\n", "#### 支持范围\n", "\n", "当前JIT Fallback特性主要应用于常量场景，即要求在编译期间能够确定实际值；有限支持部分变量场景。JIT Fallback特性还在持续完善中，下面列举出当前通过该特性已经支持的静态图编译语法。\n", "\n", "#### 创建和使用Tensor\n", "\n", "JIT Fallback支持在静态图模式下创建和使用[Tensor](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.Tensor.html)。\n", "\n", "代码用例如下，用例中的`Tensor(1, dtype=mstype.int32)`是通过JIT Fallback支持的。" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T10:51:24.264847Z", "start_time": "2022-01-04T10:51:23.515957Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1]" ] } ], "source": [ "import mindspore.nn as nn\n", "import mindspore as ms\n", "\n", "class Net(nn.Cell):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", "\n", " def construct(self):\n", " return ms.Tensor(1, dtype=ms.int32)\n", "\n", "ms.set_context(mode=ms.GRAPH_MODE)\n", "\n", "net = Net()\n", "print(net())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 调用第三方库\n", "\n", "JIT Fallback支持在静态图模式下调用第三方库的对象和方法。\n", "\n", "需要说明的是，对于具有返回值的方法，需要使用变量来保存其结果，否则可能出现报错。这个用法将在后续版本中支持。\n", "\n", "调用第三方库的代码用例如下。用例调用了NumPy第三方库，其中`np.array([1, 2, 3])`和`np.array([4, 5, 6])`是通过JIT Fallback支持的。\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T10:51:23.292278Z", "start_time": "2022-01-04T10:51:23.284465Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[5 7 9]" ] } ], "source": [ "import numpy as np\n", "import mindspore as ms\n", "import mindspore.nn as nn\n", "\n", "# pylint: disable= W0235\n", "class Net(nn.Cell):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", "\n", " def construct(self):\n", " a = np.array([1, 2, 3])\n", " b = np.array([4, 5, 6])\n", " c = a + b\n", " return ms.Tensor(c)\n", "\n", "ms.set_context(mode=ms.GRAPH_MODE)\n", "net = Net()\n", "print(net())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 使用Python原生的print打印\n", "\n", "JIT Fallback支持在静态图模式下使用Python原生的print来打印常量，它与[Print算子](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/ops/mindspore.ops.Print.html)打印信息的时机有所不同。Python原生print是在编译过程中触发打印（编译时阶段打印），而Print算子是需要图编译完成后，下发到设备端运行才打印（运行时阶段打印）。\n", "\n", "为了便于理解，举例如下。tensor_sum涉及Tensor相加，即运行时阶段才能得到结果，在调用print时，实际调用的是静态图模式中的Print算子，参考[静态图语法支持](https://www.mindspore.cn/docs/zh-CN/r2.0/note/static_graph_syntax_support.html)。而np_num是由两个NumPy常量相加得到的结果，即通过JIT Fallback支持的用法，因此在调用print时，使用的是Python原生print。由于两者的打印时机不同，最终导致显示np_sum在tensor_sum之前，即通过JIT Fallback支持的Python原生print的打印结果会在Print算子之前。\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T10:51:23.292278Z", "start_time": "2022-01-04T10:51:23.284465Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "np_sum: [2 4 6 8 10]\n", "tensor_sum: (2, 4, 6, 8, 10)" ] } ], "source": [ "import numpy as np\n", "import mindspore as ms\n", "import mindspore.nn as nn\n", "\n", "# pylint: disable= W0235\n", "class Net(nn.Cell):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", "\n", " def construct(self):\n", " x = ms.Tensor(np.array([1, 2, 3, 4, 5]))\n", " y = ms.Tensor(np.array([1, 2, 3, 4, 5]))\n", " tensor_sum = x + y\n", " print(\"tensor_sum: \", tensor_sum)\n", " x = np.array([1, 2, 3, 4, 5])\n", " y = np.array([1, 2, 3, 4, 5])\n", " np_sum = x + y\n", " print(\"np_sum: \", np_sum)\n", " return tensor_sum, ms.Tensor(np_sum)\n", "\n", "ms.set_context(mode=ms.GRAPH_MODE)\n", "net = Net()\n", "net()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "当前不支持使用同一个print同时打印编译时期和运行时期执行的信息，例如将np_sum和tensor_sum放在同一个print中将会报错。错误的代码用例如下：\n", "\n", "```python\n", "\n", "import numpy as np\n", "import mindspore as ms\n", "import mindspore.nn as nn\n", "\n", "class Net(nn.Cell):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", "\n", " def construct(self, input_x, input_y):\n", " tensor_sum = input_x + input_y\n", " x = np.array([1, 2, 3, 4, 5])\n", " y = np.array([1, 2, 3, 4, 5])\n", " np_sum = x + y\n", " print(\"np_sum: \", np_sum, \"tensor_sum: \", tensor_sum)\n", " return tensor_sum, ms.Tensor(np_sum)\n", "\n", "ms.set_context(mode=ms.GRAPH_MODE)\n", "x = ms.Tensor(np.array([1, 2, 3, 4, 5]))\n", "y = ms.Tensor(np.array([1, 2, 3, 4, 5]))\n", "net = Net()\n", "net(x,y)\n", "\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "报错信息如下:\n", "\n", "```text\n", "\n", "ValueError: When using JIT Fallback to handle script 'print(\"np_sum: \", np_sum, \"tensor_sum: \", tensor_sum)', the inputs should be constant, but found variable 'tensor_sum' to be nonconstant.\n", "\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 使用raise和assert\n", "\n", "JIT Fallback支持在静态图模式下使用raise和assert。\n", "\n", "使用raise时，要求条件语句为变量场景时，raise语句所在分支以外的分支不能返回none以外的值，否则可能出现不可预期的结果。正确的代码用例如下：\n", "\n", "```python\n", "\n", "import mindspore.nn as nn\n", "import mindspore as ms\n", "\n", "class Net(nn.Cell):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", "\n", " def construct(self, x):\n", " if x <= 0:\n", " raise ValueError(\"x should be greater than 0.\")\n", " else:\n", " x += 1\n", " return x\n", "\n", "ms.set_context(mode=ms.GRAPH_MODE)\n", "net = Net()\n", "net(-1)\n", "\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "输出结果:\n", "\n", "```text\n", "\n", "ValueError: x should be greater than 0.\n", "\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "同理，使用assert时，也需要符合常量场景的条件。正确的代码用例如下：\n", "\n", "```python\n", "\n", "import mindspore.nn as nn\n", "import mindspore as ms\n", "\n", "class Net(nn.Cell):\n", " def __init__(self):\n", " super(Net, self).__init__()\n", "\n", " def construct(self):\n", " x = 1\n", " assert 1 in [2, 3, 4]\n", " return x\n", "\n", "ms.set_context(mode=ms.GRAPH_MODE)\n", "net = Net()\n", "net()\n", "\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "输出结果中正常出现:\n", "\n", "```text\n", "\n", "AssertionError.\n", "\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 调用Python内置函数\n", "\n", "MindSpore在静态图模式下已经支持了一些Python内置函数，包括但不限于len、isinstance、map、zip等，详情请参考[静态图语法支持](https://www.mindspore.cn/docs/zh-CN/r2.0/note/static_graph_syntax_support.html)。通过JIT Fallback，可以在常量场景中支持更多的Python内置函数的用法。下面简单举例支持的部分Python内置函数。\n", "\n", "##### dict()\n", "\n", "功能：用于创建一个字典。\n", "\n", "有效输入：字典的 Key 只支持 String 类型，Value 只支持常量，不支持自定义类。\n", "\n", "暂不支持对 `dict()` 创建的字典进行循环遍历，包括 `dict.keys()`、`dict.values()`、`dict.items()`。\n", "\n", "代码用例如下：\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T10:51:23.292278Z", "start_time": "2022-01-04T10:51:23.284465Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "a: {}\n", "b: {'a': 'a', 'b': 'b', 't': 't'}\n", "c: {'one': 1, 'two': 2, 'three': 3}\n", "d: {'one': 1, 'two': 2, 'three': 3}\n" ] } ], "source": [ "import mindspore as ms\n", "\n", "@ms.jit\n", "def func():\n", " a = dict() # 创建空字典\n", " b = dict(a='a', b='b', t='t') # 传入关键字\n", " c = dict(zip(['one', 'two', 'three'], [1, 2, 3])) # 映射函数方式来构造字典\n", " d = dict([('one', 1), ('two', 2), ('three', 3)]) # 可迭代对象方式来构造字典\n", " return a, b, c, d\n", "\n", "a, b, c, d = func()\n", "print(\"a: \", a)\n", "print(\"b: \", b)\n", "print(\"c: \", c)\n", "print(\"d: \", d)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### type()\n", "\n", "功能：输出入参的类型。\n", "\n", "有效输入：Number、list、tuple、dict、np.array、常量Tensor。\n", "\n", "代码用例如下：\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T10:51:23.292278Z", "start_time": "2022-01-04T10:51:23.284465Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "a: \n", "b: \n", "c: \n", "d: \n", "e: \n", "f: \n", "g: \n" ] } ], "source": [ "import numpy as np\n", "import mindspore as ms\n", "\n", "@ms.jit\n", "def func():\n", " a = type(1)\n", " b = type(1.0)\n", " c = type([1, 2, 3])\n", " d = type((1, 2, 3))\n", " e = type({'a': 1, 'b': 2})\n", " f = type(np.array([1, 2, 3]))\n", " g = type(ms.Tensor([1, 2, 3]))\n", " return a, b, c, d, e, f, g\n", "\n", "a, b, c, d, e, f, g = func()\n", "print(\"a: \", a)\n", "print(\"b: \", b)\n", "print(\"c: \", c)\n", "print(\"d: \", d)\n", "print(\"e: \", e)\n", "print(\"f: \", f)\n", "print(\"g: \", g)\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "> type作为Python的原生函数还有另外一种使用方法，即type(name, bases, dict)返回name类型的类对象，由于该用法应用场景较少，因此暂不支持。" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### 支持常量场景下控制流\n", "\n", "为了提高Python标准语法支持度，在常量场景下实现动静统一，通过JIT Fallback实现常量场景下控制流语句的使用。控制流语句是指if、for、while等流程控制语句。JIT Fallback特性已经支持在静态图模式下创建和使用Tensor，支持调用Numpy等第三方库创建使用常量以及支持部分Python内置函数。理论上，通过JIT Fallback支持的常量语法，在常量控制流场景中也支持。\n", "代码用例如下：\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T10:51:23.292278Z", "start_time": "2022-01-04T10:51:23.284465Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "res: 2\n" ] } ], "source": [ "import numpy as np\n", "import mindspore as ms\n", "\n", "@ms.jit\n", "def func():\n", " x = np.array(1)\n", " if x <= 1:\n", " x += 1\n", " return ms.Tensor(x)\n", "\n", "res = func()\n", "print(\"res: \", res)\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### 支持运行时(Runtime)阶段的JIT Fallback\n", "\n", "JIT Fallback处理不支持的语法表达式时，将会生成相应的节点，常量会在编译时阶段推导出值，否则这些节点将传递到后端运行时，在后端通过Python的能力执行得到结果。示例代码如下，`np.add(x, y)`会生成相应节点，作为函数的返回值将会传递到运行时，当前已支持部分场景下的运行时阶段的JIT Fallback。\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ 2 4 6 8 10]\n" ] } ], "source": [ "import numpy as np\n", "import mindspore as ms\n", "\n", "@ms.jit\n", "def test_np_add():\n", " x = np.array([1, 2, 3, 4, 5])\n", " y = np.array([1, 2, 3, 4, 5])\n", " return np.add(x, y)\n", "\n", "np_add_res = test_np_add()\n", "print(np_add_res)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### 顶层图支持返回list、dict、scalar、none等基础类型\n", "\n", "##### 顶层图支持返回list\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1, \"a\", True, None, Tensor([2])]\n" ] } ], "source": [ "import mindspore as ms\n", "\n", "@ms.jit\n", "def test_return_list():\n", " return [1, \"a\", True, None, ms.Tensor([2])]\n", "\n", "res = test_return_list()\n", "print(res)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "##### 顶层图支持返回dict" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'a': ms.Tensor(np.array(1), ms.int64)}\n" ] } ], "source": [ "import mindspore as ms\n", "\n", "@ms.jit\n", "def test_return_dict():\n", " x = {'a': 1, 'b': 2}\n", " y = x.get('a')\n", " y_tensor = ms.Tensor([y])\n", " z = dict(a=y_tensor)\n", " return z\n", "\n", "res = test_return_dict()\n", "print(res)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "##### 顶层图支持返回scalar" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3\n" ] } ], "source": [ "import mindspore as ms\n", "\n", "@ms.jit\n", "def test_return_scalar(x, y):\n", " return x + y\n", "\n", "res = test_return_scalar(ms.mutable(1), ms.mutable(2))\n", "print(res)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "##### 顶层图支持返回None" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(1, 'a', None)\n" ] } ], "source": [ "import mindspore as ms\n", "\n", "@ms.jit\n", "def test_return_none():\n", " return 1, \"a\", None\n", "\n", "res = test_return_none()\n", "print(res)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "#### 使用须知\n", "\n", "在使用JIT Fallback时，请注意以下几点：\n", "\n", "1.当前JIT Fallback支持常量场景，即要求编译期间能够确定实际值。有限支持部分变量场景。\n", "\n", "2.JIT Fallback对标动态图的支持能力，须在动态图语法范围内，包括但不限于数据类型等。\n", "\n", "3.当前常量控制流场景中暂不支持对Numpy Array数据的取下标赋值，错误的代码用例如下：\n", "\n", "```python\n", "\n", "import numpy as np\n", "import mindspore as ms\n", "\n", "@ms.jit\n", "def func():\n", " x = np.array([1, 2, 3])\n", " x[0] += 1\n", " return ms.Tensor(x)\n", "\n", "res = func()\n", "print(\"res: \", res)\n", "\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "报错信息如下:\n", "\n", "```text\n", "\n", "RuntimeError: The 'setitem' operation does not support the type [External, Int64, Int64].\n", "\n", "```" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "4.值得注意的是，在常量场景中，NumPy整型数据、浮点型数据的运算结果将转换为常量进行保存，因此其运算结果可以作为函数返回值。例如：\n" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "ExecuteTime": { "end_time": "2022-01-04T10:51:23.292278Z", "start_time": "2022-01-04T10:51:23.284465Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "res: 3.0\n" ] } ], "source": [ "import numpy as np\n", "import mindspore as ms\n", "\n", "@ms.jit\n", "def test_np_add_constant():\n", " x = 1.0\n", " y = 2.0\n", " return np.add(x, y)\n", "\n", "res = test_np_add_constant()\n", "print(\"res:\", res)\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "5.通过JIT Fallback支持的NumPy第三方库，与MindSpore提供的[mindspore.numpy](https://mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore.numpy.html)不同。\n", "\n", "mindspore.numpy是通过MindSpore框架的算子能力实现的，涉及运行时阶段的算子计算，无法在编译期阶段推导其结果(变量的推导结果为None)。示例代码如下，对`mnp.average(x)`的结果使用Tensor()方法，不符合常量场景的条件，将会引发报错。\n", "\n", "```python\n", "\n", "import mindspore as ms\n", "import mindspore.numpy as mnp\n", "\n", "@ms.jit\n", "def test_mnp_average():\n", " x = mnp.array(([[1., 2.], [3., 4.]]))\n", " x_average = mnp.average(x)\n", " return ms.Tensor(x_average)\n", "\n", "out = test_mnp_average()\n", "print(out)\n", "\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "报错信息如下:\n", "\n", "```text\n", "\n", "TypeError: For 'Tensor', the type of input_data should be one of '['Tensor', 'ndarray', 'str_', 'list', 'tuple', 'float', 'int', 'bool', 'complex']', but got 'None' with type 'NoneType'.\n", "\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "MindSpore", "language": "python", "name": "mindspore" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.5" } }, "nbformat": 4, "nbformat_minor": 4 }