Differences with torch.utils.data.distributed.DistributedSampler
torch.utils.data.distributed.DistributedSampler
class torch.utils.data.distributed.DistributedSampler(dataset, num_replicas=None, rank=None, shuffle=True, seed=0, drop_last=False)
For more information, see torch.utils.data.distributed.DistributedSampler.
mindspore.dataset.DistributedSampler
class mindspore.dataset.DistributedSampler(num_shards, shard_id, shuffle=True, num_samples=None, offset=-1)
For more information, see mindspore.dataset.DistributedSampler.
Differences
PyTorch: Split datasets into shards for distributed training. The drop_last
parameter can be used to control whether to drop the tail of the data to make it evenly divisible across the devices, seed
controls the random seed of shuffle.
MindSpore: Split datasets into shards for distributed training. Drop data or make it evenly divisible across the devices is not supported, random seed is not supported to specified for shuffle.
Categories |
Subcategories |
PyTorch |
MindSpore |
Difference |
---|---|---|---|---|
Parameter |
Parameter1 |
dataset |
- |
Dataset object to be sampled. MindSpore does not need this parameter. |
Parameter2 |
num_replicas |
num_shards |
- |
|
Parameter3 |
rank |
shard_id |
- |
|
Parameter4 |
shuffle |
shuffle |
- |
|
Parameter5 |
seed |
- |
Sampling seed when |
|
Parameter6 |
drop_last |
- |
Controls whether to drop the tail of the data to make it evenly divisible across the devices. |
|
Parameter7 |
- |
num_samples |
Used to obtain partial samples. |
|
Parameter8 |
- |
offset |
The starting shard ID where the elements in the dataset are sent to. |
Code Example
import torch
from torch.utils.data.distributed import DistributedSampler
class MyMapDataset(torch.utils.data.Dataset):
def __init__(self):
super(MyMapDataset).__init__()
self.data = [i for i in range(4)]
def __getitem__(self, index):
return self.data[index]
def __len__(self):
return len(self.data)
ds = MyMapDataset()
sampler = DistributedSampler(ds, num_replicas=2, rank=1, shuffle=False)
dataloader = torch.utils.data.DataLoader(ds, sampler=sampler)
for data in dataloader:
print(data)
# Out:
# tensor([1])
# tensor([3])
import mindspore as ms
from mindspore.dataset import DistributedSampler
class MyMapDataset():
def __init__(self):
super(MyMapDataset).__init__()
self.data = [i for i in range(4)]
def __getitem__(self, index):
return self.data[index]
def __len__(self):
return len(self.data)
ds = MyMapDataset()
sampler = DistributedSampler(num_shards=2, shard_id=1, shuffle=False)
dataloader = ms.dataset.GeneratorDataset(ds, column_names=["data"], sampler=sampler)
for data in dataloader:
print(data)
# Out:
# [Tensor(shape=[], dtype=Int64, value= 1)]
# [Tensor(shape=[], dtype=Int64, value= 3)]