Function mindspore::dataset::SQuAD

Function Documentation

inline std::shared_ptr<SQuADDataset> mindspore::dataset::SQuAD(const std::string &dataset_dir, const std::string &usage = "all", int64_t num_samples = 0, ShuffleMode shuffle = ShuffleMode::kGlobal, int32_t num_shards = 1, int32_t shard_id = 0, const std::shared_ptr<DatasetCache> &cache = nullptr)

Function to create a SQuADDataset.

Note

The generated dataset has four columns [“context”, “question”, “text”, “answer_start”].

Parameters

  • dataset_dir – [in] Path to the root directory that contains the dataset.

  • usage – [in] Part of dataset of SQuAD, can be “train”, “dev” or “all” (Default=”all”).

  • num_samples – [in] The number of samples to be included in the dataset (Default=0, means all samples).

  • shuffle – [in] The mode for shuffling data every epoch (Default=ShuffleMode.kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples.

  • num_shards – [in] Number of shards that the dataset should be divided into (Default=1).

  • shard_id – [in] The shard ID within num_shards. This argument should be specified only when num_shards is also specified (Default=0).

  • cache – [in] Tensor cache to use (default=nullptr, which means no cache is used).

Returns

Shared pointer to the SQuADDataset.

Example
/* Define dataset path and MindData object */
std::string folder_path = "/path/to/squad_dataset_directory";
std::shared_ptr<Dataset> ds = SQuAD(folder_path, "train", 0, ShuffleMode::kFalse,
                                    std::make_shared<SequentialSampler>(0, 6));

/* Create iterator to read dataset */
std::shared_ptr<Iterator> iter = ds->CreateIterator();
std::unordered_map<std::string, mindspore::MSTensor> row;
iter->GetNextRow(&row);

/* Note: In SQuAD dataset, each dictionary has keys "context", "question", "text", "answer_start" */
auto context = row["context"];