mindspore.dataset.text.transforms.RegexTokenizer

class mindspore.dataset.text.transforms.RegexTokenizer(delim_pattern, keep_delim_pattern='', with_offsets=False)[source]

Tokenize a scalar tensor of UTF-8 string by regex expression pattern.

See https://unicode-org.github.io/icu/userguide/strings/regexp.html for supported regex pattern.

Note

RegexTokenizer is not supported on Windows platform yet.

Parameters

  • delim_pattern (str) – The pattern of regex delimiters. The original string will be split by matched elements.

  • keep_delim_pattern (str, optional) – The string matched by ‘delim_pattern’ can be kept as a token if it can be matched by ‘keep_delim_pattern’. The default value is an empty str which means that delimiters will not be kept as an output token (default=’’).

  • with_offsets (bool, optional) – Whether or not output offsets of tokens(default=False).

Examples

>>> # If with_offsets=False, default output is one column {["text", dtype=str]}
>>> delim_pattern = r"[ |,]"
>>> tokenizer_op = text.RegexTokenizer(delim_pattern, with_offsets=False)
>>> text_file_dataset = text_file_dataset.map(operations=tokenizer_op)
>>> # If with_offsets=True, then output three columns {["token", dtype=str],
>>> #                                                   ["offsets_start", dtype=uint32],
>>> #                                                   ["offsets_limit", dtype=uint32]}
>>> tokenizer_op = text.RegexTokenizer(delim_pattern, with_offsets=True)
>>> text_file_dataset_1 = text_file_dataset_1.map(operations=tokenizer_op, input_columns=["text"],
...                                               output_columns=["token", "offsets_start",
...                                                               "offsets_limit"],
...                                               column_order=["token", "offsets_start",
...                                                             "offsets_limit"])