Class UnicodeCharTokenizer

• Defined in File text.h

Inheritance Relationships

Base Type

• public mindspore::dataset::TensorTransform (Class TensorTransform)

Class Documentation

class UnicodeCharTokenizer : public mindspore::dataset::TensorTransform

Tokenize a scalar tensor of UTF-8 string to Unicode characters.

Public Functions

explicit UnicodeCharTokenizer(bool with_offsets = false)

Constructor.

Parameters

with_offsets – [in] whether to output offsets of tokens (default=false).

~UnicodeCharTokenizer() = default

Destructor.