mindspore::dataset::text

Classes

• Class AddToken

• Class BasicTokenizer

• Class BertTokenizer

• Class CaseFold

• Class FilterWikipediaXML

• Class JiebaTokenizer

• Class Lookup

• Class Ngram

• Class NormalizeUTF8

• Class RegexReplace

• Class RegexTokenizer

• Class SentencePieceTokenizer

• Class SlidingWindow

• Class ToNumber

• Class ToVectors

• Class Truncate

• Class TruncateSequencePair

• Class UnicodeCharTokenizer

• Class UnicodeScriptTokenizer

• Class WhitespaceTokenizer

• Class WordpieceTokenizer