mindspore.dataset.text.RegexTokenizer
- class mindspore.dataset.text.RegexTokenizer(delim_pattern, keep_delim_pattern='', with_offsets=False)[source]
Tokenize a scalar tensor of UTF-8 string by regex expression pattern.
See https://unicode-org.github.io/icu/userguide/strings/regexp.html for supported regex pattern.
Note
RegexTokenizer is not supported on Windows platform yet.
- Parameters
delim_pattern (str) – The pattern of regex delimiters. The original string will be split by matched elements.
keep_delim_pattern (str, optional) – The string matched by ‘delim_pattern’ can be kept as a token if it can be matched by ‘keep_delim_pattern’. The default value is an empty str which means that delimiters will not be kept as an output token. Default:
''
.with_offsets (bool, optional) – Whether to output the start and end offsets of each token in the original string. Default:
False
.
- Raises
- Supported Platforms:
CPU
Examples
>>> import mindspore.dataset as ds >>> import mindspore.dataset.text as text >>> >>> # Use the transform in dataset pipeline mode >>> numpy_slices_dataset = ds.NumpySlicesDataset(data=['Welcome |, To |, BeiJing!'], ... column_names=["text"]) >>> >>> # 1) If with_offsets=False, default output is one column {["text", dtype=str]} >>> delim_pattern = r"[ |,]" >>> tokenizer_op = text.RegexTokenizer(delim_pattern, with_offsets=False) >>> numpy_slices_dataset = numpy_slices_dataset.map(operations=tokenizer_op) >>> for item in numpy_slices_dataset.create_dict_iterator(num_epochs=1, output_numpy=True): ... print(item["text"]) ['Welcome' 'To' 'BeiJing!'] >>> >>> # 2) If with_offsets=True, then output three columns {["token", dtype=str], >>> # ["offsets_start", dtype=uint32], >>> # ["offsets_limit", dtype=uint32]} >>> numpy_slices_dataset = ds.NumpySlicesDataset(data=['Welcome |, To |, BeiJing!'], ... column_names=["text"]) >>> tokenizer_op = text.RegexTokenizer(delim_pattern, with_offsets=True) >>> numpy_slices_dataset = numpy_slices_dataset.map( ... operations=tokenizer_op, ... input_columns=["text"], ... output_columns=["token", "offsets_start", "offsets_limit"]) >>> for item in numpy_slices_dataset.create_dict_iterator(num_epochs=1, output_numpy=True): ... print(item["token"], item["offsets_start"], item["offsets_limit"]) ['Welcome' 'To' 'BeiJing!'] [ 0 13 21] [ 7 15 29] >>> >>> # Use the transform in eager mode >>> data = 'Welcome To BeiJing!' >>> output = text.RegexTokenizer(delim_pattern="To", keep_delim_pattern="To", with_offsets=True)(data) >>> print(output) (array(['Welcome ', 'To', ' BeiJing!'], dtype='<U12'), array([ 0, 12, 14], dtype=uint32), array([12, 14, 25], dtype=uint32))
- Tutorial Examples: