Class SentencePieceTokenizer

Inheritance Relationships

Base Type

Class Documentation

class SentencePieceTokenizer : public mindspore::dataset::TensorTransform

Tokenize a scalar token or a 1-D token to tokens by sentencepiece.

Public Functions

SentencePieceTokenizer(const std::shared_ptr<SentencePieceVocab> &vocab, mindspore::dataset::SPieceTokenizerOutType out_type)

Constructor.

Parameters

  • vocab – [in] a SentencePieceVocab object.

  • out_type – [in] The type of the output.

Example
/* Define operations */
std::shared_ptr<Dataset> ds_vocab = TextFile({"/path/to/vocab/file"}, 0, ShuffleMode::kFalse);
std::shared_ptr<SentencePieceVocab> vocab =
    ds_vocab->BuildSentencePieceVocab({}, 0, 0.9995, SentencePieceModel::kUnigram, {});
auto tokenizer_op = text::SentencePieceTokenizer(vocab, mindspore::dataset::SPieceTokenizerOutType::kString);

/* dataset is an instance of Dataset object */
dataset = dataset->Map({tokenizer_op},   // operations
                       {"text"});        // input columns
inline SentencePieceTokenizer(const std::string &vocab_path, mindspore::dataset::SPieceTokenizerOutType out_type)

Constructor.

Parameters

  • vocab_path – [in] vocab model file path.

  • out_type – [in] The type of the output.

Example
/* Define operations */
auto tokenizer_op = text::SentencePieceTokenizer("/path/to/model",
                                                 mindspore::dataset::SPieceTokenizerOutType::kInt);

/* dataset is an instance of Dataset object */
dataset = dataset->Map({tokenizer_op},   // operations
                       {"text"});        // input columns
SentencePieceTokenizer(const std::vector<char> &vocab_path, mindspore::dataset::SPieceTokenizerOutType out_type)

Constructor.

Parameters

  • vocab_path – [in] vocab model file path. type should be char of vector.

  • out_type – [in] The type of the output.

~SentencePieceTokenizer() = default

Destructor.