mindspore.dataset.text.SentencePieceModel

View Source On Gitee
class mindspore.dataset.text.SentencePieceModel[source]

Subword algorithms for SentencePiece.

Available values are as follows:

  • SentencePieceModel.UNIGRAM: Unigram Language Model subword algorithm.

  • SentencePieceModel.BPE: Byte-Pair-Encoding subword algorithm.

  • SentencePieceModel.CHAR: Character-based subword algorithm.

  • SentencePieceModel.WORD: Word-based subword algorithm.