Class IWSLT2016Dataset

Inheritance Relationships

Base Type

Class Documentation

class IWSLT2016Dataset : public mindspore::dataset::Dataset

A source dataset for reading and parsing IWSLT2016 dataset.

Public Functions

IWSLT2016Dataset(const std::vector<char> &dataset_dir, const std::vector<char> &usage, const std::vector<std::vector<char>> &language_pair, const std::vector<char> &valid_set, const std::vector<char> &test_set, int64_t num_samples, ShuffleMode shuffle, int32_t num_shards, int32_t shard_id, const std::shared_ptr<DatasetCache> &cache)

Constructor of IWSLT2016Dataset.

Note

The generated dataset has two columns [“text”, “translation”].

Parameters

  • dataset_dir – [in] Path to the root directory that contains the dataset.

  • usage – [in] Part of dataset of IWSLT2016, can be “train”, “valid”, “test” or “all”.

  • language_pair – [in] List containing src and tgt language.

  • valid_set – [in] A string to identify validation set.

  • test_set – [in] A string to identify test set.

  • num_samples – [in] The number of samples to be included in the dataset.

  • shuffle – [in] The mode for shuffling data every epoch. Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples.

  • num_shards – [in] Number of shards that the dataset should be divided into.

  • shard_id – [in] The shard ID within num_shards. This argument should be specified only when num_shards is also specified.

  • cache – [in] Tensor cache to use.

~IWSLT2016Dataset() override = default

Destructor of IWSLT2016Dataset.