mindspore.dataset.sync_wait_for_dataset
- mindspore.dataset.sync_wait_for_dataset(rank_id, rank_size, current_epoch)[source]
Wait util the dataset files required by all devices are downloaded.
Note
It should be used together with mindspore.dataset.OBSMindDataset and be called before each epoch.
- Parameters
Examples
>>> # Create a synchronization callback >>> import mindspore as ms >>> from mindspore.dataset import sync_wait_for_dataset >>> >>> class SyncForDataset(ms.Callback): ... def __init__(self): ... super(SyncForDataset, self).__init__() ... def epoch_begin(self, run_context): ... cb_params = run_context.original_args() ... epoch_num = cb_params.cur_epoch_num ... sync_wait_for_dataset(rank_id, rank_size, epoch_num)