mindspore.dataset.text.UnicodeCharTokenizer
- class mindspore.dataset.text.UnicodeCharTokenizer(with_offsets=False)[source]
Tokenize a scalar tensor of UTF-8 string to Unicode characters.
- Parameters
with_offsets (bool, optional) – Whether or not output offsets of tokens. Default:
False
.- Raises
TypeError – If with_offsets is not of type bool.
- Supported Platforms:
CPU
Examples
>>> import mindspore.dataset as ds >>> import mindspore.dataset.text as text >>> >>> # If with_offsets=False, default output one column {["text", dtype=str]} >>> tokenizer_op = text.UnicodeCharTokenizer(with_offsets=False) >>> text_file_dataset = text_file_dataset.map(operations=tokenizer_op) >>> # If with_offsets=True, then output three columns {["token", dtype=str], ["offsets_start", dtype=uint32], >>> # ["offsets_limit", dtype=uint32]} >>> tokenizer_op = text.UnicodeCharTokenizer(with_offsets=True) >>> >>> text_file_list = ["/path/to/text_file_dataset_file"] >>> text_file_dataset = ds.TextFileDataset(dataset_files=text_file_list) >>> text_file_dataset = text_file_dataset.map(operations=tokenizer_op, input_columns=["text"], ... output_columns=["token", "offsets_start", "offsets_limit"])
- Tutorial Examples: