mindspore.dataset.text.transforms.UnicodeCharTokenizer
- class mindspore.dataset.text.transforms.UnicodeCharTokenizer(with_offsets=False)[source]
Tokenize a scalar tensor of UTF-8 string to Unicode characters.
- Parameters
with_offsets (bool, optional) – Whether or not output offsets of tokens (default=False).
- Raises
TypeError – If with_offsets is not of type bool.
- Supported Platforms:
CPU
Examples
>>> # If with_offsets=False, default output one column {["text", dtype=str]} >>> tokenizer_op = text.UnicodeCharTokenizer(with_offsets=False) >>> text_file_dataset = text_file_dataset.map(operations=tokenizer_op) >>> # If with_offsets=True, then output three columns {["token", dtype=str], ["offsets_start", dtype=uint32], >>> # ["offsets_limit", dtype=uint32]} >>> tokenizer_op = text.UnicodeCharTokenizer(with_offsets=True) >>> text_file_dataset = text_file_dataset.map(operations=tokenizer_op, input_columns=["text"], ... output_columns=["token", "offsets_start", "offsets_limit"], ... column_order=["token", "offsets_start", "offsets_limit"])