Process Control Statements
Currently, there are two execution modes of a mainstream deep learning framework: a static graph mode (GRAPH_MODE
) and a dynamic graph mode (PYNATIVE_MODE
).
In PYNATIVE_MODE
, MindSpore fully supports process control statements of the native Python syntax. In GRAPH_MODE
, MindSpore performance is optimized during build. Therefore, there are some special constraints on using process control statements when during network definition. Other constraints are the same as those in the native Python syntax.
When switching the running mode from dynamic graph to static graph, pay attention to the static graph syntax support. The following describes how to use process control statements when defining a network in GRAPH_MODE
.
Constant and Variable Conditions
When a network is defined in GRAPH_MODE
, MindSpore classifies condition expressions in process control statements into constant and variable conditions. During graph build, a condition expression that can be determined to be either true or false is a constant condition, while a condition expression that cannot be determined to be true or false is a variable condition. MindSpore generates control flow operators on a network only when the condition expression is a variable condition.
It should be noted that, when a control flow operator exists in a network, the network is divided into multiple execution subgraphs, and process jumping and data transmission between the subgraphs cause performance loss to some extent.
Constant Conditions
Check methods:
The condition expression does not contain tensors or any list, tuple, or dict whose elements are of the tensor type.
The condition expression contains tensors or a list, tuple, or dict of the tensor type, but the condition expression result is not affected by the tensor value.
Examples:
for i in range(0,10)
:i
is a scalar. The result of the potential condition expressioni < 10
can be determined during graph build. Therefore, it is a constant condition.self.flag
: A scalar of the Boolean type. Its value is determined when the Cell object is built.x + 1 < 10
:x
is a scalar: The value ofx + 1
is uncertain when the Cell object is built. MindSpore computes the results of all scalar expressions during graph build. Therefore, the value of the expression is determined during build.len(my_list) < 10
:my_list
is a list object whose element is of the tensor type. This condition expression contains tensors, but the expression result is not affected by the tensor value and is related only to the number of tensors inmy_list
.
Variable Conditions
Check method:
The condition expression contains tensors or a list, tuple, or dict of the tensor type, and the condition expression result is affected by the tensor value.
Examples:
x < y
:x
andy
are operator outputs.x in list
:x
is the operator output.
The operator output can be determined only when each step is executed. Therefore, the preceding two conditions are variable conditions.
if Statement
When defining a network in GRAPH_MODE
using the if
statement, pay attention to the following: When the condition expression is a variable condition, the same variable in different branches must be assigned the same data type.
if Statement Under a Variable Condition
In the following code, shapes of tensors assigned to the out
variable in the if
and else
branches are ()
and (2,)
, respectively. The shape of the tensor returned by the network is determined by the condition x < y
. The result of x < y
cannot be determined during graph build. Therefore, whether the out
shape is ()
or (2,)
cannot be determined during graph build. MindSpore throws an exception due to type derivation failure.
import numpy as np
import mindspore as ms
from mindspore import nn
class SingleIfNet(nn.Cell):
def construct(self, x, y, z):
# Build an if statement whose condition expression is a variable condition.
if x < y:
out = x
else:
out = z
out = out + 1
return out
forward_net = SingleIfNet()
x = ms.Tensor(np.array(0), dtype=ms.int32)
y = ms.Tensor(np.array(1), dtype=ms.int32)
z = ms.Tensor(np.array([1, 2]), dtype=ms.int32)
output = forward_net(x, y, z)
Execute the preceding code. The error information is as follows:
ValueError: mindspore/ccsrc/pipeline/jit/static_analysis/static_analysis.cc:800 ProcessEvalResults] Cannot join the return values of different branches, perhaps you need to make them equal.
Shape Join Failed: shape1 = (), shape2 = (2).
if Statement Under a Constant Condition
When the condition expression in the if
statement is a constant condition, the usage of the condition expression is the same as that of the native Python syntax, and there is no additional constraint. In the following code, the condition expression x < y + 1
of the if
statement is a constant condition (because x
and y
are scalar constants). During graph build, the out
variable is of the scalar int
type. The network can be built and executed properly, and the correct result 1
is displayed.
import numpy as np
import mindspore as ms
from mindspore import nn
class SingleIfNet(nn.Cell):
def construct(self, z):
x = 0
y = 1
# Build an if statement whose condition expression is a constant condition.
if x < y + 1:
out = x
else:
out = z
out = out + 1
return out
z = ms.Tensor(np.array([0, 1]), dtype=ms.int32)
forward_net = SingleIfNet()
output = forward_net(z)
print("output:", output)
output: 1
for Statement
The for
statement expands the loop body. Therefore, the number of subgraphs and operators of the network that uses the for
statement depends on the number of loops of the for
statement. If the number of operators or subgraphs is too large, more hardware resources are consumed.
In the following sample code, the loop body in the for
statement is executed for three times, and the output is 5
.
import numpy as np
from mindspore import nn
import mindspore as ms
class IfInForNet(nn.Cell):
def construct(self, x, y):
out = 0
# Build a for statement whose condition expression is a constant condition.
for i in range(0, 3):
# Build an if statement whose condition expression is a variable condition.
if x + i < y:
out = out + x
else:
out = out + y
out = out + 1
return out
forward_net = IfInForNet()
x = ms.Tensor(np.array(0), dtype=ms.int32)
y = ms.Tensor(np.array(1), dtype=ms.int32)
output = forward_net(x, y)
print("output:", output)
output: 5
The for
statement expands the loop body. Therefore, the preceding code is equivalent to the following code:
import numpy as np
from mindspore import nn
import mindspore as ms
class IfInForNet(nn.Cell):
def construct(self, x, y):
out = 0
# Loop: 0
if x + 0 < y:
out = out + x
else:
out = out + y
out = out + 1
# Loop: 1
if x + 1 < y:
out = out + x
else:
out = out + y
out = out + 1
# Loop: 2
if x + 2 < y:
out = out + x
else:
out = out + y
out = out + 1
return out
forward_net = IfInForNet()
x = ms.Tensor(np.array(0), dtype=ms.int32)
y = ms.Tensor(np.array(1), dtype=ms.int32)
output = forward_net(x, y)
print("output:", output)
output: 5
According to the preceding sample code, using the for
statement may cause too many subgraphs in some scenarios. To reduce hardware resource overhead and improve network build performance, you can convert the for
statement to the while
statement whose condition expression is a variable condition.
while Statement
The while
statement is more flexible than the for
statement. When the condition of while
is a constant, while
processes and expands the loop body in a similar way as for
.
When the condition expression of while
is a variable condition, the while
statement does not expand the loop body. Instead, a control flow operator is generated during graph execution. Therefore, the problem of too many subgraphs caused by the for
loop can be avoided.
while Statement Under a Constant Condition
In the following sample code, the loop body in the for
statement is executed for three times, and the output result is 5
, which is essentially the same as the sample code in the for
statement.
import numpy as np
from mindspore import nn
import mindspore as ms
class IfInWhileNet(nn.Cell):
def construct(self, x, y):
i = 0
out = x
# Build a while statement whose condition expression is a constant condition.
while i < 3:
# Build an if statement whose condition expression is a variable condition.
if x + i < y:
out = out + x
else:
out = out + y
out = out + 1
i = i + 1
return out
forward_net = IfInWhileNet()
x = ms.Tensor(np.array(0), dtype=ms.int32)
y = ms.Tensor(np.array(1), dtype=ms.int32)
output = forward_net(x, y)
print("output:", output)
output: 5
while Statement Under a Variable Condition
Constraint 1: When the condition expression in the while statement is a variable condition, the while loop body cannot contain computation operations of non-tensor types, such as scalar, list, and tuple.
To avoid too many control flow operators, you can use the
while
statement whose condition expression is a variable condition to rewrite the preceding code.import numpy as np from mindspore import nn import mindspore as ms class IfInWhileNet(nn.Cell): def construct(self, x, y, i): out = x # Build a while statement whose condition expression is a variable condition. while i < 3: # Build an if statement whose condition expression is a variable condition. if x + i < y: out = out + x else: out = out + y out = out + 1 i = i + 1 return out forward_net = IfInWhileNet() i = ms.Tensor(np.array(0), dtype=ms.int32) x = ms.Tensor(np.array(0), dtype=ms.int32) y = ms.Tensor(np.array(1), dtype=ms.int32) output = forward_net(x, y, i) print("output:", output)
output: 5
It should be noted that in the preceding code, the condition expression of the
while
statement is a variable condition, and thewhile
loop body is not expanded. The expressions in thewhile
loop body are computed during the running of each step. In addition, the following constraints are generated:When the condition expression in the
while
statement is a variable condition, thewhile
loop body cannot contain computation operations of non-tensor types, such as scalar, list, and tuple.These types of computation operations are completed during graph build, which conflicts with the computation mechanism of the
while
loop body during execution. The following uses sample code as an example:import numpy as np from mindspore import nn import mindspore as ms class IfInWhileNet(nn.Cell): def __init__(self): super().__init__() self.nums = [1, 2, 3] def construct(self, x, y, i): j = 0 out = x # Build a while statement whose condition expression is a variable condition. while i < 3: if x + i < y: out = out + x else: out = out + y out = out + self.nums[j] i = i + 1 # Build scalar computation in the loop body of the while statement whose condition expression is a variable condition. j = j + 1 return out forward_net = IfInWhileNet() i = ms.Tensor(np.array(0), dtype=ms.int32) x = ms.Tensor(np.array(0), dtype=ms.int32) y = ms.Tensor(np.array(1), dtype=ms.int32) output = forward_net(x, y, i)
In the preceding code, the
while
loop body of the condition expressioni < 3
contains scalar computationj = j + 1
. As a result, an error occurs during graph build. The following error information is displayed during code execution:IndexError: mindspore/core/abstract/prim_structures.cc:127 InferTupleOrListGetItem] list_getitem evaluator index should be in range[-3, 3), but got 3.
Constraint 2: When the condition expression in the while statement is a variable condition, the input shape of the operator cannot be changed in the loop body.
MindSpore requires that the input shape of the same operator on the network be determined during graph build. However, changing the input shape of the operator in the
while
loop body takes effect during graph execution.The following uses sample code as an example:
import numpy as np from mindspore import nn import mindspore as ms from mindspore import ops class IfInWhileNet(nn.Cell): def __init__(self): super().__init__() self.expand_dims = ops.ExpandDims() def construct(self, x, y, i): out = x # Build a while statement whose condition expression is a variable condition. while i < 3: if x + i < y: out = out + x else: out = out + y out = out + 1 # Change the input shape of an operator. out = self.expand_dims(out, -1) i = i + 1 return out forward_net = IfInWhileNet() i = ms.Tensor(np.array(0), dtype=ms.int32) x = ms.Tensor(np.array(0), dtype=ms.int32) y = ms.Tensor(np.array(1), dtype=ms.int32) output = forward_net(x, y, i)
In the preceding code, the
ExpandDims
operator in thewhile
loop body of the condition expressioni < 3
changes the input shape of the expressionout = out + 1
in the next loop. As a result, an error occurs during graph build. The following error information is displayed during code execution:ValueError: mindspore/ccsrc/pipeline/jit/static_analysis/static_analysis.cc:800 ProcessEvalResults] Cannot join the return values of different branches, perhaps you need to make them equal. Shape Join Failed: shape1 = (1), shape2 = (1, 1).