Differences with torch.distributed.all_gather

torch.distributed.all_gather

torch.distributed.all_gather(
    tensor_list,
    tensor,
    group=None,
    async_op=False
)

For more information, see torch.distributed.all_gather.

mindspore.ops.AllGather

class mindspore.ops.AllGather(group=GlobalComm.WORLD_COMM_GROUP)(input_x)

For more information, see mindspore.ops.AllGather.

Differences

PyTorch: The inputs are the tensor broadcasted by the current process tensor, the communication group group and the async op flag async_op. The output is tensor_list after AllGather op, whose type is list[Tensor] and the length is the number of devices in the communication group. The return is a async work handle if async_op=True, otherwise is None.

MindSpore: This interface input is tensor input_x and the output is tensor. The first dimension is the number of devices N in the communication domain, and the rest of the dimensions is the same as the input tensor, rather than outputting list[Tensor] as the PyTorch interface does. This interface currently does not support the configuration of async_op.

Class	Sub-class	PyTorch	MindSpore	Difference
Parameters	Parameter 1	tensor_list	-	PyTorch: the output after AllGather. MindSpore does not have this parameter
	Parameter 2	tensor	-	PyTorch: the tensor broadcasted by the current process. MindSpore does not have this parameter
	Parameter 3	group	group	-
	Parameter 4	async_op	-	PyTorch: the async op flag. MindSpore does not have this parameter
Input	Single input	-	input_x	PyTorch: not applied. MindSpore: the input tensor of AllGather.