Usage Constraints During Operator Parallel

View Source On Gitee

API name

constraints

mindspore.ops.Abs

None

mindspore.ops.ACos

None

mindspore.ops.Acosh

None

mindspore.ops.Add

None

mindspore.ops.AddN

None

mindspore.ops.ApproximateEqual

None

mindspore.ops.ArgMaxWithValue

When the input_x is splited on the axis dimension, the distributed result may be inconsistent with that on the single machine.

mindspore.ops.ArgMinWithValue

When the input_x is splited on the axis dimension, the distributed result may be inconsistent with that on the single machine.

mindspore.ops.Asin

None

mindspore.ops.Asinh

None

mindspore.ops.Assign

None

mindspore.ops.AssignAdd

None

mindspore.ops.AssignSub

None

mindspore.ops.Atan

None

mindspore.ops.Atan2

None

mindspore.ops.Atanh

None

mindspore.ops.AvgPool

1. The data format only supports ‘NCHW’;
2. The shapes of output H/W dimension must be divisible by the split strategies of input H/W dimension;
3. If H/W is split:
1) If the kernel_size <= stride, the input slice size must be divisible by stride;
2) It does not support kernel_size > stride;
4. In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.AvgPool3D

1. The data format only supports ‘NCDHW’;
2. If data exchange between adjacent nodes is involved, only Ascend is supported;
3. The W dimensions can not be split;
4. The output shape of D/H dimension must be divisible by the strategy of input D/H dimensions;
5. In valid mode: If D/H dimension is split:
1) When the kernel_size <= stride, the input‘s slice shape of D/H dimension must be divisible by stride;
2) It does not support that kernel_size > stride;
6. In the same/pad mode: If D/H dimension is split:
1) If kernel_size >= stride, (Total input length including pad - kernel_size) must be divisible by stride. Otherwise, the pad must be 0 and the slice shape of D/H dimension must be divisible by stride;
2) (Output length* stride - input length) must be divisible by strategy:
3) The length of data sent and received between adjacent cards must be greater than or equal to 0 and less than or equal to the slice size;
7. In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.BatchMatMul

None

mindspore.ops.BatchNorm

It does not support GPU.

mindspore.ops.BesselI0e

None

mindspore.ops.BesselI1e

None

mindspore.ops.BiasAdd

None

mindspore.ops.BitwiseAnd

None

mindspore.ops.BitwiseOr

None

mindspore.ops.BitwiseXor

None

mindspore.ops.BoundingBoxEncode

1. The first dimension of input (anchor_box) and input (groundtruth_box) can be split;
2. The sharding strategies of input (anchor_box) and input (groundtruth_box) must be the same.

mindspore.ops.BroadcastTo

None

mindspore.ops.Cast

The shard strategy is ignored in the Auto Parallel and Semi Auto Parallel mode.

mindspore.ops.Cdist

1. The strategy for ‘B’ dimension must be the same;
2.M dimension can’t be split.

mindspore.ops.Ceil

None

mindspore.ops.Concat

The input_x can’t be split into the dimension of axis, otherwise it’s inconsistent with the single machine in the mathematical logic.

mindspore.ops.Conv2D

1. The data format only supports ‘NCHW’;
2. If data exchange between adjacent nodes is involved, only Ascend is supported;
3. When the value of group is not 1, can not split C-in or C-out;
4. The last two dimensions of weight can not be split;
5. The output shape of H/W dimension must be divisible by the strategy of input H/W dimensions;
6. In valid mode: If H/W dimension is split:
1) When the kernel_size <= stride (kernel_size is dilation (kernel_size - 1) + 1, the same below), the input‘s slice shape of H/W dimension must be divisible by stride;
2) It does not support that kernel_size > stride;
7. In the same/pad mode: If H/W dimension is split:
1) If kernel_size >= stride, (Total input length including pad - kernel_size) must be divisible by stride. Otherwise, the pad must be 0 and the slice shape of H/W dimension must be divisible by stride;
2) (Output length
stride - input length) must be divisible by strategy:
3) The length of data sent and received between adjacent cards must be greater than or equal to 0 and less than or equal to the slice size;

mindspore.ops.Conv3D

1. The data format only supports ‘NCDHW’;
2. If data exchange between adjacent nodes is involved, only Ascend is supported;
3. When the value of group is not 1, can not split C-in or C-out;
4. The W dimension and the last three dimensions of weight can not be split;
5. The output shape of D/H dimension must be divisible by the strategy of input D/H dimensions;
6. In valid mode: If D/H dimension is split:
1) When the kernel_size <= stride (kernel_size is dilation (kernel_size - 1) + 1, the same below), the input‘s slice shape of D/H dimension must be divisible by stride;
2) It does not support that kernel_size > stride;
7. In the same/pad mode: If D/H dimension is split:
1) If kernel_size >= stride, (Total input length including pad - kernel_size) must be divisible by stride. Otherwise, the pad must be 0 and the slice shape of D/H dimension must be divisible by stride;
2) (Output length
stride - input length) must be divisible by strategy:
3) The length of data sent and received between adjacent cards must be greater than or equal to 0 and less than or equal to the slice size;
8. In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.Cos

None

mindspore.ops.Cosh

None

mindspore.ops.CropAndResize

1. Sharding of the H/W dimension of input (x) and the second dimension of input (boxes) is not supported.
2. The shard strategy for the first dimension of inputs (boxes) and (box_index) must be the same.

mindspore.ops.CumProd

The axis dimension for input can’t be split.

mindspore.ops.CumSum

The same as CumProd.

mindspore.ops.Div

None

mindspore.ops.DivNoNan

None

mindspore.ops.Dropout

None

mindspore.ops.Elu

None

mindspore.ops.EmbeddingLookup

The same as Gather.

mindspore.ops.Equal

None

mindspore.ops.Erf

None

mindspore.ops.Erfc

None

mindspore.ops.Erfinv

None

mindspore.ops.Exp

None

mindspore.ops.ExpandDims

None

mindspore.ops.Expm1

None

mindspore.ops.Floor

None

mindspore.ops.FloorDiv

None

mindspore.ops.FloorMod

None

mindspore.ops.Gamma

1. Set the strategy for shape. e.g shape=(8, 16), the corresponding policy can be (2, 4);
2. The strategy for alpha and beta must be all-1;
3. When the setting for shard is not all-1 strategy, the result is inconsistent with standalone.

mindspore.ops.Gather

1. When batch_dims > 0:
1) The axis dimension of input_params can not be split;
2) Non-uniform split is not supported;
2. When batch_dims = 0:
1) If uniform split:
a) and if the axis dimensions of input_params is not split, other dimensions can be split;
b) If the axis dimensions of input_params is split: The input_params only supports 1 and 2 dimensions; The input_indices can not be scalar tensor; Does not support to split input_params and input_indices at the same time; When axis = 0 and the parameter is split in the dimension of axis, the output strategy can be configured. The legal output shard strategy is (indices_strategy, param_strategy[1:]) or ((indices_strategy[0]*param_strategy[0], indices_strategy[1:]), param_strategy[1:])
2) Non-uniform split:
a) Only support axis = 0;
b) The non-uniform split only represents the non-uniformity of the 0th dimension of input_params, and the last dimension of the params slice should be aligned by 32 bytes;
c) The number of slices in the 0th dimension of input_params should be equal to that of the last dimension of input_indices;
d) Each dimension of input_params can be split, but input_indices can only split the last dimension, and does not support repeated calculations;
e) Input_indices shall meet the following requirements: the Tensor value of the next slice shall be greater than that of the previous slice.

mindspore.ops.GatherD

The dimension corresponding to dim cannot be segmented; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.GatherNd

The first input can’t be split, and the last dimension of the second input can’t be split; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.GeLU

None

mindspore.ops.Greater

None

mindspore.ops.GreaterEqual

None

mindspore.ops.HShrink

None

mindspore.ops.HSigmoid

None

mindspore.ops.InplaceAdd

The first dimension of x and input_v can’t be split.

mindspore.ops.InplaceSub

The same as InplaceAdd.

mindspore.ops.InplaceUpdate

The same as InplaceAdd.

mindspore.ops.Inv

None

mindspore.ops.IOU

The first dimension of the anchor_boxes and gt_boxes can be spilt.

mindspore.ops.IsFinite

None

mindspore.ops.KLDivLoss

None

mindspore.ops.L2Loss

None

mindspore.ops.L2Normalize

The input_x can’t be split into the dimension of axis, otherwise it’s inconsistent with the single machine in the mathematical logic.

mindspore.ops.Lerp

None

mindspore.ops.Less

None

mindspore.ops.LessEqual

None

mindspore.ops.LinSpace

You don’t need to configure strategy for start and end. You just need to pass in a strategy of length 1 whose value divisible into num.

mindspore.ops.LogicalAnd

None

mindspore.ops.LogicalNot

None

mindspore.ops.LogicalOr

None

mindspore.ops.Log

None

mindspore.ops.Log1p

None

mindspore.ops.LogSoftmax

The logits can’t be split into the dimension of axis, otherwise it’s inconsistent with the single machine in the mathematical logic.

mindspore.ops.MaskedFill

None

mindspore.ops.MatMul

1. When transpose_b=True is set, the input’s split strategy must be in the form of ((A, B), (C, B));
2. When transpose_b=False is set, the input’s split strategy must be in the form of ((A, B), (B, C));
3. It is supported to set the output’s split strategy, the legal output’s split strategy is ((A, C),) or ((A * B, C),)

mindspore.ops.Maximum

None

mindspore.ops.MaxPool

1. The data format only supports ‘NCHW’;
2. The shapes of output H/W dimension must be divisible by the split strategies of input H/W dimension;
3. If H/W is split:
1) If the kernel_size <= stride, the input slice size must be divisible by stride;
2) It does not support kernel_size > stride;
4. In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.MaxPool3D

The same as AvgPool3D.

mindspore.ops.Minimum

None

mindspore.ops.Mish

None

mindspore.ops.Mod

None

mindspore.ops.Mul

None

mindspore.ops.MulNoNan

None

mindspore.ops.Neg

None

mindspore.ops.NotEqual

None

mindspore.ops.OneHot

Only support 1-dim indices. Must configure strategy for the output and the first and second inputs.

mindspore.ops.OnesLike

None

mindspore.ops.Pow

None

mindspore.ops.PReLU

When the shape of weight is not [1], the shard strategy in channel dimension of input_x should be consistent with weight.

mindspore.ops.RandomChoiceWithMask

Only the all-1 strategy is supported.

mindspore.ops.RealDiv

None

mindspore.ops.Reciprocal

None

mindspore.ops.ReduceMax

When the input_x is splited on the axis dimension, the distributed result may be inconsistent with that on the single machine.

mindspore.ops.ReduceMin

When the input_x is splited on the axis dimension, the distributed result may be inconsistent with that on the single machine.

mindspore.ops.ReduceSum

None

mindspore.ops.ReduceMean

None

mindspore.ops.ReLU

None

mindspore.ops.ReLU6

None

mindspore.ops.Reshape

Configuring shard strategy is not supported. In auto parallel mode, if multiple operators are followed by the reshape operator, different shard strategys are not allowed to be configured for these operators.

mindspore.ops.ResizeBilinear

Under GPU platform, can not split H or W dimension; Under Ascend platform, can not split H dimension, and the output shape of W dimension can be divided by the strategy.

mindspore.ops.Rint

None

mindspore.ops.ResizeNearestNeighbor

When align_corners=True is set, only the first dimension and the second dimension can be split.

mindspore.ops.ROIAlign

Sharding the H/W dimension of the input(features) and the second dimension of input(rois) is not supported.

mindspore.ops.Round

None

mindspore.ops.Rsqrt

None

mindspore.ops.ScatterAdd

The second input cannot be split, and the top n dimensions of the third input (n is the dimension of the second input) cannot be split; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.ScatterDiv

The second input cannot be split, and the top n dimensions of the third input (n is the dimension of the second input) cannot be split; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.ScatterMax

The first dimension of the first input cannot be split, the second input cannot be split, and the top n dimensions of the third input (n is the dimension of the second input) cannot be split; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.ScatterMin

The first dimension of the first input cannot be split, the second input cannot be split, and the top n dimensions of the third input (n is the dimension of the second input) cannot be split; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.ScatterMul

The second input cannot be split, and the top n dimensions of the third input (n is the dimension of the second input) cannot be split; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.ScatterNdAdd

The second input cannot be split, the top n-1 dimension of the third input (n is the dimension of the second input) cannot be split, and the remaining k dimensions (excluding the top n-1 dimension) of the third input are consistent with the last k partitions of the first input; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.ScatterNdSub

The second input cannot be split, the top n-1 dimension of the third input (n is the dimension of the second input) cannot be split, and the remaining k dimensions (excluding the top n-1 dimension) of the third input are consistent with the last k partitions of the first input; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.ScatterNdUpdate

The top m dimension of the first input cannot be cut (m is the value of the last dimension of the second input indexes [- 1]). The second input cannot be split. The top n-1 dimension of the third input (n is the dimension of the second input) cannot be split. The partitions of the remaining k dimensions (excluding the top n-1 dimension) of the third input is consistent with the last k partitions of the first input; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.ScatterSub

The second input cannot be split, and the top n dimensions of the third input (n is the dimension of the second input) cannot be split; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.ScatterUpdate

The first dimension of first input can not be split, the second input can not be split, and the first n dimensions (n is the dimension size of the second input) of the third input can not be split; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.TensorScatterAdd

The second input cannot be split, the top n-1 dimension of the third input (n is the dimension of the second input) cannot be split, and the remaining k dimensions (excluding the top n-1 dimension) of the third input are consistent with the last k partitions of the first input; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.TensorScatterDiv

The second input cannot be split, the top n-1 dimension of the third input (n is the dimension of the second input) cannot be split, and the remaining k dimensions (excluding the top n-1 dimension) of the third input are consistent with the last k partitions of the first input; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.TensorScatterMax

The top m dimension of the first input cannot be cut (m is the value of the last dimension of the second input indexes [- 1]). The second input cannot be split. The top n-1 dimension of the third input (n is the dimension of the second input) cannot be split. The partitions of the remaining k dimensions (excluding the top n-1 dimension) of the third input is consistent with the last k partitions of the first input; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.TensorScatterMax

The top m dimension of the first input cannot be cut (m is the value of the last dimension of the second input indexes [- 1]). The second input cannot be split. The top n-1 dimension of the third input (n is the dimension of the second input) cannot be split. The partitions of the remaining k dimensions (excluding the top n-1 dimension) of the third input is consistent with the last k partitions of the first input; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.TensorScatterMul

The second input cannot be split, the top n-1 dimension of the third input (n is the dimension of the second input) cannot be split, and the remaining k dimensions (excluding the top n-1 dimension) of the third input are consistent with the last k partitions of the first input; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.TensorScatterAdd

The second input cannot be split, the top n-1 dimension of the third input (n is the dimension of the second input) cannot be split, and the remaining k dimensions (excluding the top n-1 dimension) of the third input are consistent with the last k partitions of the first input; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.TensorScatterUpdate

The top m dimension of the first input cannot be cut (m is the value of the last dimension of the second input indexes [- 1]). The second input cannot be split. The top n-1 dimension of the third input (n is the dimension of the second input) cannot be split. The partitions of the remaining k dimensions (excluding the top n-1 dimension) of the third input is consistent with the last k partitions of the first input; In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.Select

In auto_parallel mode, the dual recursive algorithm is not supported.

mindspore.ops.SeLU

None

mindspore.ops.Sigmoid

None

mindspore.ops.SigmoidCrossEntropyWithLogits

None

mindspore.ops.Sign

None

mindspore.ops.Sin

None

mindspore.ops.Sinh

None

mindspore.ops.Softmax

The logits can’t be split into the dimension of axis, otherwise it’s inconsistent with the single machine in the mathematical logic.

mindspore.ops.SoftmaxCrossEntropyWithLogits

The last dimension of logits and labels can’t be splited; Only supports using output[0].

mindspore.ops.Softplus

None

mindspore.ops.Softsign

None

mindspore.ops.SoftShrink

None

mindspore.ops.SparseGatherV2

The same as Gather.

mindspore.ops.Split

The input_x can’t be split into the dimension of axis, otherwise it’s inconsistent with the single machine in the mathematical logic.

mindspore.ops.Sqrt

None

mindspore.ops.Square

None

mindspore.ops.SquaredDifference

None

mindspore.ops.Squeeze

None

mindspore.ops.Stack

None

mindspore.ops.StridedSlice

Only support mask with all 0 values; The dimension needs to be split should be all extracted; Split is supported when the strides of dimension is 1.

mindspore.ops.Slice

The dimension needs to be split should be all extracted.

mindspore.ops.Sub

None

mindspore.ops.Tan

None

mindspore.ops.Tanh

None

mindspore.ops.Tile

Only support configuring shard strategy for multiples.

mindspore.ops.TopK

The input_x can’t be split into the last dimension, otherwise it’s inconsistent with the single machine in the mathematical logic.

mindspore.ops.Transpose

None

mindspore.ops.TruncateDiv

None

mindspore.ops.TruncateMod

None

mindspore.ops.Unique

Only support the repeat calculate shard strategy (1,).

mindspore.ops.UnsortedSegmentSum

The shard of input_x and segment_ids must be the same as the dimension of segment_ids.

mindspore.ops.UnsortedSegmentMin

The shard of input_x and segment_ids must be the same as the dimension of segment_ids. Note that if the segment id i is missing, then the output[i] will be filled with the maximum of the input type. The user needs to mask the maximum value to avoid value overflow. The communication operation such as AllReudce will raise an Run Task Error due to overflow.

mindspore.ops.UnsortedSegmentMax

The shard of input_x and segment_ids must be the same as the dimension of segment_ids. Note that if the segment id i is missing, then the output[i] will be filled with the minimum of the input type. The user needs to mask the minimum value to avoid value overflow. The communication operation such as AllReudce will raise an Run Task Error due to overflow.

mindspore.ops.Xdivy

None

mindspore.ops.Xlogy

None

mindspore.ops.ZerosLike

None

Repeated calculation means that the device is not fully used. For example, the cluster has 8 devices to run distributed training, the splitting strategy only cuts the input into 4 copies. In this case, double counting will occur.