Class WhitespaceTokenizer

• Defined in File text.h

Inheritance Relationships

Base Type

• public mindspore::dataset::TensorTransform (Class TensorTransform)

Class Documentation

class WhitespaceTokenizer : public mindspore::dataset::TensorTransform

Tokenize a scalar tensor of UTF-8 string on ICU4C defined whitespaces.

Public Functions

explicit WhitespaceTokenizer(bool with_offsets = false)

Constructor.

Example

/* Define operations */
auto tokenizer_op = text::WhitespaceTokenizer(false);

/* dataset is an instance of Dataset object */
dataset = dataset->Map({tokenizer_op},   // operations
                       {"text"});        // input columns

Parameters

with_offsets – [in] whether to output offsets of tokens (default=false).

~WhitespaceTokenizer() = default

Destructor.