mindspore.dataset.text.transforms.UnicodeScriptTokenizer

class mindspore.dataset.text.transforms.UnicodeScriptTokenizer(keep_whitespace=False, with_offsets=False)[source]

Tokenize a scalar tensor of UTF-8 string on Unicode script boundaries.

Note

UnicodeScriptTokenizer is not supported on Windows platform yet.

Parameters

keep_whitespace (bool, optional) – If or not emit whitespace tokens (default=False).
with_offsets (bool, optional) – If or not output offsets of tokens (default=False).

Examples

>>> # If with_offsets=False, default output one column {["text", dtype=str]}
>>> tokenizer_op = text.UnicodeScriptTokenizer(keep_whitespace=True, with_offsets=False)
>>> text_file_dataset = text_file_dataset.map(operations=tokenizer_op)
>>> # If with_offsets=False, then output three columns {["token", dtype=str],
>>> #                                                   ["offsets_start", dtype=uint32],
>>> #                                                   ["offsets_limit", dtype=uint32]}
>>> tokenizer_op = text.UnicodeScriptTokenizer(keep_whitespace=True, with_offsets=True)
>>> text_file_dataset = text_file_dataset.map(operations=tokenizer_op, input_columns=["text"],
...                                           output_columns=["token", "offsets_start", "offsets_limit"],
...                                           column_order=["token", "offsets_start", "offsets_limit"])