Class CLUEDataset

Inheritance Relationships

Base Type

Class Documentation

class CLUEDataset : public mindspore::dataset::Dataset

A source dataset for reading and parsing CLUE dataset.

Public Functions

CLUEDataset(const std::vector<std::vector<char>> &dataset_files, const std::vector<char> &task, const std::vector<char> &usage, int64_t num_samples, ShuffleMode shuffle, int32_t num_shards, int32_t shard_id, const std::shared_ptr<DatasetCache> &cache)

Constructor of CLUEDataset.

Parameters

  • dataset_files – [in] List of files to be read to search for a pattern of files. The list will be sorted in a lexicographical order.

  • task – [in] The kind of task, one of “AFQMC”, “TNEWS”, “IFLYTEK”, “CMNLI”, “WSC” and “CSL” (default=”AFQMC”).

  • usage – [in] Part of dataset of CLUE, can be “train”, “test” or “eval” data (default=”train”).

  • num_samples – [in] The number of samples to be included in the dataset (Default = 0 means all samples).

  • shuffle – [in] The mode for shuffling data every epoch. (Default=ShuffleMode.kGlobal) Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples.

  • num_shards – [in] Number of shards that the dataset should be divided into. (Default = 1)

  • shard_id – [in] The shard ID within num_shards. This argument should be specified only when num_shards is also specified (Default = 0).

  • cache – [in] Tensor cache to use (default=nullptr which means no cache is used).

~CLUEDataset() override = default

Destructor of CLUEDataset.