Class UnicodeCharTokenizer
Defined in File text.h
Inheritance Relationships
Base Type
public mindspore::dataset::TensorTransform
(Class TensorTransform)
Class Documentation
-
class UnicodeCharTokenizer : public mindspore::dataset::TensorTransform
Tokenize a scalar tensor of UTF-8 string to Unicode characters.
Public Functions
-
explicit UnicodeCharTokenizer(bool with_offsets = false)
Constructor.
- Parameters
with_offsets – [in] whether to output offsets of tokens (default=false).
样例/* Define operations */ auto tokenizer_op = text::UnicodeCharTokenizer(); /* dataset is an instance of Dataset object */ dataset = dataset->Map({tokenizer_op}, // operations {"text"}); // input columns
-
~UnicodeCharTokenizer() = default
Destructor.
-
explicit UnicodeCharTokenizer(bool with_offsets = false)