{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 动静结合\n",
"\n",
"[](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r1.8/tutorials/zh_cn/advanced/pynative_graph/mindspore_combine.ipynb) \n",
"[](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r1.8/tutorials/zh_cn/advanced/pynative_graph/mindspore_combine.py) \n",
"[](https://gitee.com/mindspore/docs/blob/r1.8/tutorials/source_zh_cn/advanced/pynative_graph/combine.ipynb)\n",
"\n",
"当前在业界支持动态图和静态图两种模式，动态图通过解释执行，具有动态语法亲和性，表达灵活；静态图使用JIT(just in time)编译优化执行，偏静态语法，在语法上有较多限制。动态图和静态图的编译流程不一致，导致语法约束也不一致。\n",
"\n",
"MindSpore针对动态图和静态图模式，首先统一API表达，在两种模式下使用相同的API；其次统一动态图和静态图的底层微分机制。\n",
"\n",
"\n",
"\n",
"## 实现原理\n",
"\n",
"MindSpore支持使用`ms_function`装饰器来修饰需要用静态图执行的对象，从而实现动静结合的目的。下面我们通过一个简单的动静结合的示例来介绍其实现原理。示例代码如下："
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"init x:\n",
" [[1. 1. 1.]\n",
" [1. 1. 1.]\n",
" [1. 1. 1.]]\n",
"\n",
"x:\n",
" [[1024. 1024. 1024.]\n",
" [1024. 1024. 1024.]\n",
" [1024. 1024. 1024.]]\n"
]
}
],
"source": [
"import numpy as np\n",
"import mindspore.nn as nn\n",
"import mindspore as ms\n",
"from mindspore import ms_function\n",
"\n",
"class Add(nn.Cell):\n",
" \"\"\"自定义类实现x自身相加\"\"\"\n",
" def construct(self, x):\n",
" x = x + x\n",
" x = x + x\n",
" return x\n",
"\n",
"class Mul(nn.Cell):\n",
" \"\"\"自定义类实现x自身相乘\"\"\"\n",
" @ms_function # 使用ms_function修饰，此函数以静态图方式执行\n",
" def construct(self, x):\n",
" x = x * x\n",
" x = x * x\n",
" return x\n",
"\n",
"class Test(nn.Cell):\n",
" \"\"\"自定义类实现x先Add(x)，后Mul(x)，再Add(x)\"\"\"\n",
" def __init__(self):\n",
" super(Test, self).__init__()\n",
" self.add = Add()\n",
" self.mul = Mul()\n",
"\n",
" def construct(self, x):\n",
" x = self.add(x)\n",
" x = self.mul(x)\n",
" x = self.add(x)\n",
" return x\n",
"\n",
"ms.set_context(mode=ms.PYNATIVE_MODE)\n",
"x = ms.Tensor(np.ones([3, 3], dtype=np.float32))\n",
"print(\"init x:\\n\", x)\n",
"net = Test()\n",
"x = net(x)\n",
"print(\"\\nx:\\n\", x)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"从上面的打印结果可以看出，经过Test运算后，x最终值为每个元素都是8的3\\*3矩阵。该用例按照执行序，编译的方式如下图所示：\n",
"\n",
"\n",
"\n",
"被`ms_function`修饰的函数将会按照静态图的方式进行编译和执行。如果网络涉及到反向求导，被`ms_function`修饰的部分也将以整图的形式来生成反向图，并与前后单个算子的反向图连成一个整体后被下发执行。其中，缓存的策略与静态图的缓存策略一致，相同的函数对象在输入Shape和Type信息一致时，编译的图结构将会被缓存。\n",
"\n",
"## `ms_function`装饰器\n",
"\n",
"为了提高动态图模式下的前向计算任务执行速度，MindSpore提供了`ms_function`装饰器，可以通过修饰Python函数或者Python类的成员函数使其被编译成计算图，通过图优化等技术提高运行速度。\n",
"\n",
"### 使用方式\n",
"\n",
"MindSpore支持在动态图下使用静态编译的方式来进行混合执行，通过使用`ms_function`装饰符来修饰需要用静态图来执行的函数对象，即可实现动态图和静态图的混合执行。\n",
"\n",
"#### 1. 修饰独立函数\n",
"\n",
"使用`ms_function`装饰器时，可以对独立定义的函数进行修饰，使其在Graph模式下运行，示例如下："
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[5. 7. 9.]\n"
]
}
],
"source": [
"import numpy as np\n",
"import mindspore.ops as ops\n",
"import mindspore as ms\n",
"from mindspore import ms_function\n",
"\n",
"# 设置运行模式为动态图模式\n",
"ms.set_context(mode=ms.PYNATIVE_MODE)\n",
"\n",
"# 使用装饰器，指定静态图模式下执行\n",
"@ms_function\n",
"def add_func(x, y):\n",
" return ops.add(x, y)\n",
"\n",
"x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n",
"y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32))\n",
"\n",
"out = add_func(x, y)\n",
"print(out)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"在上面的示例代码中，虽然一开始设置了运行模式为动态图模式，但是由于使用了`ms_function`装饰器对函数`add_func(x, y)`进行了修饰，所以函数`add_func(x, y)`仍然是以静态图模式运行。\n",
"\n",
"#### 2. 修饰Cell的成员函数\n",
"\n",
"使用`ms_function`装饰器时，可以对`Cell`的成员函数进行修饰，示例代码如下："
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Infer result:\n",
" [5. 7. 9.]\n",
"Gradient result:\n",
"Grad x Tensor1:\n",
" [1. 1. 1.]\n",
"Grad y Tensor2:\n",
" [1. 1. 1.]\n"
]
}
],
"source": [
"import numpy as np\n",
"import mindspore.nn as nn\n",
"import mindspore.ops as ops\n",
"import mindspore as ms\n",
"from mindspore import ms_function\n",
"\n",
"# 设置运行模式为动态图模式\n",
"ms.set_context(mode=ms.PYNATIVE_MODE)\n",
"\n",
"class Add(nn.Cell):\n",
"\n",
" @ms_function # 使用装饰器，指定静态图模式下执行\n",
" def construct(self, x, y):\n",
" out = x + y\n",
" return out\n",
"\n",
"x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n",
"y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32))\n",
"\n",
"grad_ops = ops.GradOperation(get_all=True) # 定义求导操作\n",
"net = Add()\n",
"grad_out = grad_ops(net)(x, y)\n",
"\n",
"print(\"Infer result:\\n\", net(x, y))\n",
"\n",
"print(\"Gradient result:\")\n",
"print(\"Grad x Tensor1:\\n\", grad_out[0]) # 对x求导\n",
"print(\"Grad y Tensor2:\\n\", grad_out[1]) # 对y求导"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"从上面的打印结果可以看出，x与y相加的结果为\\[5, 7, 9\\]， 对x求导的结果和对y求导的结果相同，都为\\[1, 1, 1\\]。\n",
"\n",
"### 注意事项\n",
"\n",
"在使用ms_function来修饰函数，加速执行效率时，请注意以下几点：\n",
"\n",
"1. ms_function修饰的函数须在静态图编译支持的语法范围内，包括但不限于数据类型等。\n",
"\n",
"2. ms_function修饰的函数所支持的控制流语法，与静态图保持一致。其中，仅对固定循环次数或者分支条件的控制流结构具有加速效果。\n",
"\n",
"3. 在PyNative模式下使用ms_function功能时，非ms_function修饰的部分支持断点调试；被ms_function修饰的部分由于是以静态图的方式编译，不支持断点调试。\n",
"\n",
"4. 由于ms_function修饰的函数将按照静态图的方式编译执行，因此ms_function不支持修饰的函数中含有Hook算子，也不支持修饰自定义Bprop函数。\n",
"\n",
"5. ms_function修饰的函数会受到静态图函数副作用的影响。函数副作用指：当调用函数时，除了函数返回值之外，还对主调用函数产生的附加影响，例如修改全局变量（函数外的变量），修改函数的参数等。\n",
"\n",
"场景1："
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"5\n"
]
}
],
"source": [
"import numpy as np\n",
"import mindspore as ms\n",
"from mindspore import ms_function\n",
"\n",
"# pylint: disable=W0612\n",
"\n",
"value = 5\n",
"\n",
"@ms_function\n",
"def func(x, y):\n",
" out = x + y\n",
" value = 1\n",
" return out\n",
"\n",
"ms.set_context(mode=ms.PYNATIVE_MODE)\n",
"x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n",
"y = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n",
"func(x, y)\n",
"print(value)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"该场景下，`value`是全局变量且在`func`函数中被修改。此时，如果用`ms_function`修饰`func`函数，全局变量`value`的值将不会被修改。原因是：**静态图编译时，会优化掉与返回值无关的语句**。\n",
"\n",
"场景2："
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"out1: [6. 7. 8.]\n",
"out2: [6. 7. 8.]\n"
]
}
],
"source": [
"import numpy as np\n",
"import mindspore.nn as nn\n",
"import mindspore as ms\n",
"from mindspore import ms_function\n",
"\n",
"class Func(nn.Cell):\n",
" def __init__(self):\n",
" super(Func, self).__init__()\n",
" self.value = 5\n",
"\n",
" @ms_function\n",
" def construct(self, x):\n",
" out = self.value + x\n",
" return out\n",
"\n",
"ms.set_context(mode=ms.PYNATIVE_MODE)\n",
"x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n",
"func = Func()\n",
"print(\"out1:\", func(x))\n",
"func.value = 1\n",
"print(\"out2:\", func(x))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"从上面的打印可以看出，在修改了Func类的成员变量value的值为1之后，对成员函数construct的操作并无影响。这是因为在此场景下，用ms_function修饰了`Func`对象的`construct`成员函数，执行`Func`时将会以静态图的方式编译执行。由于静态图会缓存编译结果，第二次调用`Func`时，对`value`的修改不会生效。\n",
"\n",
"6. 加装了`ms_function`装饰器的函数中，如果包含不需要进行参数训练的算子（如`MatMul`、`Add`等算子），则这些算子可以在被装饰的函数中直接调用；如果被装饰的函数中包含了需要进行参数训练的算子（如`Conv2D`、`BatchNorm`等算子），则这些算子必须在被装饰的函数之外完成实例化操作。下面我们通过示例代码对这两种场景进行说明。\n",
"\n",
"场景1：在被装饰的函数中直接调用不需要进行参数训练的算子（示例中为`mindspore.ops.Add`）。示例代码如下："
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"x: [1. 2. 3.] \n",
"y: [4. 5. 6.] \n",
"z: [5. 7. 9.]\n"
]
}
],
"source": [
"import numpy as np\n",
"import mindspore as ms\n",
"import mindspore.ops as ops\n",
"from mindspore import ms_function\n",
"\n",
"ms.set_context(mode=ms.PYNATIVE_MODE)\n",
"\n",
"add = ops.Add()\n",
"\n",
"@ms_function\n",
"def add_fn(x, y):\n",
" res = add(x, y)\n",
" return res\n",
"\n",
"x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n",
"y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32))\n",
"z = add_fn(x, y)\n",
"\n",
"print(\"x:\", x.asnumpy(), \"\\ny:\", y.asnumpy(), \"\\nz:\", z.asnumpy())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"场景2：需要进行参数训练的算子（示例中为`mindspore.nn.Conv2d`），必须在被装饰的函数之外完成实例化操作，示例代码如下："
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[[[ 0.00829158 -0.02994147]\n",
" [-0.09116832 -0.00181637]]\n",
"\n",
" [[-0.00519348 -0.02172063]\n",
" [-0.04015012 -0.02083161]]\n",
"\n",
" [[ 0.00608188 -0.01443425]\n",
" [-0.01468289 0.01200477]]\n",
"\n",
" [[ 0.00845292 0.00044869]\n",
" [-0.00361492 0.01993337]]]]\n"
]
}
],
"source": [
"import numpy as np\n",
"import mindspore.nn as nn\n",
"import mindspore as ms\n",
"from mindspore import ms_function\n",
"\n",
"ms.set_context(mode=ms.PYNATIVE_MODE)\n",
"\n",
"# 对函数conv_fn中的算子conv_obj完成实例化操作\n",
"conv_obj = nn.Conv2d(in_channels=3, out_channels=4, kernel_size=3, stride=2, padding=0)\n",
"conv_obj.init_parameters_data()\n",
"\n",
"@ms_function\n",
"def conv_fn(x):\n",
" res = conv_obj(x)\n",
" return res\n",
"\n",
"input_data = np.random.randn(1, 3, 3, 3).astype(np.float32)\n",
"z = conv_fn(ms.Tensor(input_data))\n",
"print(z.asnumpy())"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "MindSpore",
"language": "python",
"name": "mindspore"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
}
},
"nbformat": 4,
"nbformat_minor": 4
}