Function mindspore::dataset::IWSLT2017
Defined in File datasets.h
Function Documentation
Function to create a IWSLT2017Dataset.
Note
The generated dataset has two columns [“text”, “translation”].
- Parameters
dataset_dir – [in] Path to the root directory that contains the dataset.
usage – [in] Part of dataset of IWSLT2017, can be “train”, “valid”, “test” or “all” (default = “all”).
language_pair – [in] List containing src and tgt language (Default = {“de”, “en”}).
num_samples – [in] The number of samples to be included in the dataset. (Default = 0, means all samples).
shuffle – [in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples.
num_shards – [in] Number of shards that the dataset should be divided into (Default = 1).
shard_id – [in] The shard ID within num_shards. This argument should be specified only when num_shards is also specified (Default = 0).
cache – [in] Tensor cache to use (default=nullptr, which means no cache is used).
- Returns
Shared pointer to the IWSLT2017Dataset.