Class SentencePieceTokenizer
Defined in File text.h
Inheritance Relationships
Base Type
public mindspore::dataset::TensorTransform
(Class TensorTransform)
Class Documentation
-
class SentencePieceTokenizer : public mindspore::dataset::TensorTransform
Tokenize a scalar token or a 1-D token to tokens by sentencepiece.
Public Functions
Constructor.
- Parameters
vocab – [in] a SentencePieceVocab object.
out_type – [in] The type of the output.
样例/* Define operations */ std::shared_ptr<Dataset> ds_vocab = TextFile({"/path/to/vocab/file"}, 0, ShuffleMode::kFalse); std::shared_ptr<SentencePieceVocab> vocab = ds_vocab->BuildSentencePieceVocab({}, 0, 0.9995, SentencePieceModel::kUnigram, {}); auto tokenizer_op = text::SentencePieceTokenizer(vocab, mindspore::dataset::SPieceTokenizerOutType::kString); /* dataset is an instance of Dataset object */ dataset = dataset->Map({tokenizer_op}, // operations {"text"}); // input columns
-
inline SentencePieceTokenizer(const std::string &vocab_path, mindspore::dataset::SPieceTokenizerOutType out_type)
Constructor.
- Parameters
vocab_path – [in] vocab model file path.
out_type – [in] The type of the output.
样例/* Define operations */ auto tokenizer_op = text::SentencePieceTokenizer("/path/to/model", mindspore::dataset::SPieceTokenizerOutType::kInt); /* dataset is an instance of Dataset object */ dataset = dataset->Map({tokenizer_op}, // operations {"text"}); // input columns
-
SentencePieceTokenizer(const std::vector<char> &vocab_path, mindspore::dataset::SPieceTokenizerOutType out_type)
Constructor.
- Parameters
vocab_path – [in] vocab model file path. type should be char of vector.
out_type – [in] The type of the output.
-
~SentencePieceTokenizer() = default
Destructor.