Class MindDataDataset
Defined in File datasets.h
Inheritance Relationships
Base Type
public mindspore::dataset::Dataset
(Class Dataset)
Class Documentation
-
class MindDataDataset : public mindspore::dataset::Dataset
A source dataset for reading and parsing MindRecord dataset.
Public Functions
-
MindDataDataset(const std::vector<char> &dataset_file, const std::vector<std::vector<char>> &columns_list, const std::shared_ptr<Sampler> &sampler, const nlohmann::json *padded_sample, int64_t num_padded, ShuffleMode shuffle_mode = ShuffleMode::kGlobal, const std::shared_ptr<DatasetCache> &cache = nullptr)
Constructor of MindDataDataset.
- Parameters
dataset_file – [in] File name of one component of a mindrecord source. Other files with identical source in the same path will be found and loaded automatically.
columns_list – [in] List of columns to be read (default={}).
sampler – [in] Shared pointer to a sampler object used to choose samples from the dataset. If sampler is not given, a
RandomSampler
will be used to randomly iterate the entire dataset (default = RandomSampler()), supported sampler list: SubsetRandomSampler, PkSampler, RandomSampler, SequentialSampler, DistributedSampler.padded_sample – [in] Samples will be appended to dataset, where keys are the same as column_list.
num_padded – [in] Number of padding samples. Dataset size plus num_padded should be divisible by num_shards.
shuffle_mode – [in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples. ShuffleMode::kInfile - Shuffle samples in file.
cache – [in] Tensor cache to use (default=nullptr which means no cache is used).
-
MindDataDataset(const std::vector<char> &dataset_file, const std::vector<std::vector<char>> &columns_list, const Sampler *sampler, const nlohmann::json *padded_sample, int64_t num_padded, ShuffleMode shuffle_mode = ShuffleMode::kGlobal, const std::shared_ptr<DatasetCache> &cache = nullptr)
Constructor of MindDataDataset.
- Parameters
dataset_file – [in] File name of one component of a mindrecord source. Other files with identical source in the same path will be found and loaded automatically.
columns_list – [in] List of columns to be read.
sampler – [in] Raw pointer to a sampler object used to choose samples from the dataset. supported sampler list: SubsetRandomSampler, PkSampler, RandomSampler, SequentialSampler, DistributedSampler.
padded_sample – [in] Samples will be appended to dataset, where keys are the same as column_list.
num_padded – [in] Number of padding samples. Dataset size plus num_padded should be divisible by num_shards.
shuffle_mode – [in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples. ShuffleMode::kInfile - Shuffle samples in file.
cache – [in] Tensor cache to use (default=nullptr which means no cache is used).
-
MindDataDataset(const std::vector<char> &dataset_file, const std::vector<std::vector<char>> &columns_list, const std::reference_wrapper<Sampler> &sampler, const nlohmann::json *padded_sample, int64_t num_padded, ShuffleMode shuffle_mode = ShuffleMode::kGlobal, const std::shared_ptr<DatasetCache> &cache = nullptr)
Constructor of MindDataDataset.
- Parameters
dataset_file – [in] File name of one component of a mindrecord source. Other files with identical source in the same path will be found and loaded automatically.
columns_list – [in] List of columns to be read.
sampler – [in] Sampler object used to choose samples from the dataset. supported sampler list: SubsetRandomSampler, PkSampler, RandomSampler, SequentialSampler, DistributedSampler.
padded_sample – [in] Samples will be appended to dataset, where keys are the same as column_list.
num_padded – [in] Number of padding samples. Dataset size plus num_padded should be divisible by num_shards.
shuffle_mode – [in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples. ShuffleMode::kInfile - Shuffle samples in file.
cache – [in] Tensor cache to use (default=nullptr which means no cache is used).
-
MindDataDataset(const std::vector<std::vector<char>> &dataset_files, const std::vector<std::vector<char>> &columns_list, const std::shared_ptr<Sampler> &sampler, const nlohmann::json *padded_sample, int64_t num_padded, ShuffleMode shuffle_mode = ShuffleMode::kGlobal, const std::shared_ptr<DatasetCache> &cache = nullptr)
Constructor of MindDataDataset.
- Parameters
dataset_files – [in] List of dataset files to be read directly.
columns_list – [in] List of columns to be read.
sampler – [in] Raw pointer to a sampler object used to choose samples from the dataset. supported sampler list: SubsetRandomSampler, PkSampler, RandomSampler, SequentialSampler, DistributedSampler.
padded_sample – [in] Samples will be appended to dataset, where keys are the same as column_list.
num_padded – [in] Number of padding samples. Dataset size plus num_padded should be divisible by num_shards.
shuffle_mode – [in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples. ShuffleMode::kInfile - Shuffle data within each file.
cache – [in] Tensor cache to use (default=nullptr which means no cache is used).
-
MindDataDataset(const std::vector<std::vector<char>> &dataset_files, const std::vector<std::vector<char>> &columns_list, const Sampler *sampler, const nlohmann::json *padded_sample, int64_t num_padded, ShuffleMode shuffle_mode = ShuffleMode::kGlobal, const std::shared_ptr<DatasetCache> &cache = nullptr)
Constructor of MindDataDataset.
- Parameters
dataset_files – [in] List of dataset files to be read directly.
columns_list – [in] List of columns to be read.
sampler – [in] Raw pointer to a sampler object used to choose samples from the dataset. supported sampler list: SubsetRandomSampler, PkSampler, RandomSampler, SequentialSampler, DistributedSampler.
padded_sample – [in] Samples will be appended to dataset, where keys are the same as column_list.
num_padded – [in] Number of padding samples. Dataset size plus num_padded should be divisible by num_shards.
shuffle_mode – [in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples. ShuffleMode::kInfile - Shuffle data within each file.
cache – [in] Tensor cache to use (default=nullptr which means no cache is used).
-
MindDataDataset(const std::vector<std::vector<char>> &dataset_files, const std::vector<std::vector<char>> &columns_list, const std::reference_wrapper<Sampler> &sampler, const nlohmann::json *padded_sample, int64_t num_padded, ShuffleMode shuffle_mode = ShuffleMode::kGlobal, const std::shared_ptr<DatasetCache> &cache = nullptr)
Constructor of MindDataDataset.
- Parameters
dataset_files – [in] List of dataset files to be read directly.
columns_list – [in] List of columns to be read.
sampler – [in] Sampler object used to choose samples from the dataset. supported sampler list: SubsetRandomSampler, PkSampler, RandomSampler, SequentialSampler, DistributedSampler.
padded_sample – [in] Samples will be appended to dataset, where keys are the same as column_list.
num_padded – [in] Number of padding samples. Dataset size plus num_padded should be divisible by num_shards.
shuffle_mode – [in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples. ShuffleMode::kInfile - Shuffle samples in file.
cache – [in] Tensor cache to use (default=nullptr which means no cache is used).
-
~MindDataDataset() override = default
Destructor of MindDataDataset.
-
MindDataDataset(const std::vector<char> &dataset_file, const std::vector<std::vector<char>> &columns_list, const std::shared_ptr<Sampler> &sampler, const nlohmann::json *padded_sample, int64_t num_padded, ShuffleMode shuffle_mode = ShuffleMode::kGlobal, const std::shared_ptr<DatasetCache> &cache = nullptr)