Class MindDataDataset

Inheritance Relationships

Base Type

Class Documentation

class MindDataDataset : public mindspore::dataset::Dataset

A source dataset for reading and parsing MindRecord dataset.

Public Functions

MindDataDataset(const std::vector<char> &dataset_file, const std::vector<std::vector<char>> &columns_list, const std::shared_ptr<Sampler> &sampler, const nlohmann::json *padded_sample, int64_t num_padded, ShuffleMode shuffle_mode = ShuffleMode::kGlobal, const std::shared_ptr<DatasetCache> &cache = nullptr)

Constructor of MindDataDataset.

Parameters

  • dataset_file – [in] File name of one component of a mindrecord source. Other files with identical source in the same path will be found and loaded automatically.

  • columns_list – [in] List of columns to be read (default={}).

  • sampler – [in] Shared pointer to a sampler object used to choose samples from the dataset. If sampler is not given, a RandomSampler will be used to randomly iterate the entire dataset (default = RandomSampler()), supported sampler list: SubsetRandomSampler, PkSampler, RandomSampler, SequentialSampler, DistributedSampler.

  • padded_sample – [in] Samples will be appended to dataset, where keys are the same as column_list.

  • num_padded – [in] Number of padding samples. Dataset size plus num_padded should be divisible by num_shards.

  • shuffle_mode – [in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples. ShuffleMode::kInfile - Shuffle samples in file.

  • cache – [in] Tensor cache to use (default=nullptr which means no cache is used).

MindDataDataset(const std::vector<char> &dataset_file, const std::vector<std::vector<char>> &columns_list, const Sampler *sampler, const nlohmann::json *padded_sample, int64_t num_padded, ShuffleMode shuffle_mode = ShuffleMode::kGlobal, const std::shared_ptr<DatasetCache> &cache = nullptr)

Constructor of MindDataDataset.

Parameters

  • dataset_file – [in] File name of one component of a mindrecord source. Other files with identical source in the same path will be found and loaded automatically.

  • columns_list – [in] List of columns to be read.

  • sampler – [in] Raw pointer to a sampler object used to choose samples from the dataset. supported sampler list: SubsetRandomSampler, PkSampler, RandomSampler, SequentialSampler, DistributedSampler.

  • padded_sample – [in] Samples will be appended to dataset, where keys are the same as column_list.

  • num_padded – [in] Number of padding samples. Dataset size plus num_padded should be divisible by num_shards.

  • shuffle_mode – [in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples. ShuffleMode::kInfile - Shuffle samples in file.

  • cache – [in] Tensor cache to use (default=nullptr which means no cache is used).

MindDataDataset(const std::vector<char> &dataset_file, const std::vector<std::vector<char>> &columns_list, const std::reference_wrapper<Sampler> sampler, const nlohmann::json *padded_sample, int64_t num_padded, ShuffleMode shuffle_mode = ShuffleMode::kGlobal, const std::shared_ptr<DatasetCache> &cache = nullptr)

Constructor of MindDataDataset.

Parameters

  • dataset_file – [in] File name of one component of a mindrecord source. Other files with identical source in the same path will be found and loaded automatically.

  • columns_list – [in] List of columns to be read.

  • sampler – [in] Sampler object used to choose samples from the dataset. supported sampler list: SubsetRandomSampler, PkSampler, RandomSampler, SequentialSampler, DistributedSampler.

  • padded_sample – [in] Samples will be appended to dataset, where keys are the same as column_list.

  • num_padded – [in] Number of padding samples. Dataset size plus num_padded should be divisible by num_shards.

  • shuffle_mode – [in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples. ShuffleMode::kInfile - Shuffle samples in file.

  • cache – [in] Tensor cache to use (default=nullptr which means no cache is used).

MindDataDataset(const std::vector<std::vector<char>> &dataset_files, const std::vector<std::vector<char>> &columns_list, const std::shared_ptr<Sampler> &sampler, const nlohmann::json *padded_sample, int64_t num_padded, ShuffleMode shuffle_mode = ShuffleMode::kGlobal, const std::shared_ptr<DatasetCache> &cache = nullptr)

Constructor of MindDataDataset.

Parameters

  • dataset_files – [in] List of dataset files to be read directly.

  • columns_list – [in] List of columns to be read.

  • sampler – [in] Raw pointer to a sampler object used to choose samples from the dataset. supported sampler list: SubsetRandomSampler, PkSampler, RandomSampler, SequentialSampler, DistributedSampler.

  • padded_sample – [in] Samples will be appended to dataset, where keys are the same as column_list.

  • num_padded – [in] Number of padding samples. Dataset size plus num_padded should be divisible by num_shards.

  • shuffle_mode – [in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples. ShuffleMode::kInfile - Shuffle data within each file.

  • cache – [in] Tensor cache to use (default=nullptr which means no cache is used).

MindDataDataset(const std::vector<std::vector<char>> &dataset_files, const std::vector<std::vector<char>> &columns_list, const Sampler *sampler, const nlohmann::json *padded_sample, int64_t num_padded, ShuffleMode shuffle_mode = ShuffleMode::kGlobal, const std::shared_ptr<DatasetCache> &cache = nullptr)

Constructor of MindDataDataset.

Parameters

  • dataset_files – [in] List of dataset files to be read directly.

  • columns_list – [in] List of columns to be read.

  • sampler – [in] Raw pointer to a sampler object used to choose samples from the dataset. supported sampler list: SubsetRandomSampler, PkSampler, RandomSampler, SequentialSampler, DistributedSampler.

  • padded_sample – [in] Samples will be appended to dataset, where keys are the same as column_list.

  • num_padded – [in] Number of padding samples. Dataset size plus num_padded should be divisible by num_shards.

  • shuffle_mode – [in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples. ShuffleMode::kInfile - Shuffle data within each file.

  • cache – [in] Tensor cache to use (default=nullptr which means no cache is used).

MindDataDataset(const std::vector<std::vector<char>> &dataset_files, const std::vector<std::vector<char>> &columns_list, const std::reference_wrapper<Sampler> sampler, const nlohmann::json *padded_sample, int64_t num_padded, ShuffleMode shuffle_mode = ShuffleMode::kGlobal, const std::shared_ptr<DatasetCache> &cache = nullptr)

Constructor of MindDataDataset.

Parameters

  • dataset_files – [in] List of dataset files to be read directly.

  • columns_list – [in] List of columns to be read.

  • sampler – [in] Sampler object used to choose samples from the dataset. supported sampler list: SubsetRandomSampler, PkSampler, RandomSampler, SequentialSampler, DistributedSampler.

  • padded_sample – [in] Samples will be appended to dataset, where keys are the same as column_list.

  • num_padded – [in] Number of padding samples. Dataset size plus num_padded should be divisible by num_shards.

  • shuffle_mode – [in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples. ShuffleMode::kInfile - Shuffle samples in file.

  • cache – [in] Tensor cache to use (default=nullptr which means no cache is used).

~MindDataDataset() = default

Destructor of MindDataDataset.