mindspore.dataset.text.WhitespaceTokenizer

class mindspore.dataset.text.WhitespaceTokenizer(with_offsets=False)[source]

Tokenize a scalar tensor of UTF-8 string on ICU4C defined whitespaces, such as: ‘ ‘, ‘\t’, ‘\r’, ‘\n’.

Note

WhitespaceTokenizer is not supported on Windows platform yet.

Parameters

with_offsets (bool, optional) – Whether to output the start and end offsets of each token in the original string. Default: False .

Raises

TypeError – If with_offsets is not of type bool.

Supported Platforms:

CPU

Examples

>>> import mindspore.dataset as ds
>>> import mindspore.dataset.text as text
>>>
>>> # Use the transform in dataset pipeline mode
>>> numpy_slices_dataset = ds.NumpySlicesDataset(data=['Welcome     To   BeiJing!'], column_names=["text"])
>>>
>>> # 1) If with_offsets=False, default output one column {["text", dtype=str]}
>>> tokenizer_op = text.WhitespaceTokenizer(with_offsets=False)
>>> numpy_slices_dataset = numpy_slices_dataset.map(operations=tokenizer_op)
>>> for item in numpy_slices_dataset.create_dict_iterator(num_epochs=1, output_numpy=True):
...     print(item["text"])
['Welcome' 'To' 'BeiJing!']
>>>
>>> # 2) If with_offsets=True, then output three columns {["token", dtype=str],
>>> #                                                     ["offsets_start", dtype=uint32],
>>> #                                                     ["offsets_limit", dtype=uint32]}
>>> numpy_slices_dataset = ds.NumpySlicesDataset(data=['Welcome     To   BeiJing!'], column_names=["text"])
>>> tokenizer_op = text.WhitespaceTokenizer(with_offsets=True)
>>> numpy_slices_dataset = numpy_slices_dataset.map(
...     operations=tokenizer_op,
...     input_columns=["text"],
...     output_columns=["token", "offsets_start", "offsets_limit"])
>>> for item in numpy_slices_dataset.create_dict_iterator(num_epochs=1, output_numpy=True):
...     print(item["token"], item["offsets_start"], item["offsets_limit"])
['Welcome' 'To' 'BeiJing!'] [ 0 12 17] [ 7 14 25]
>>>
>>> # Use the transform in eager mode
>>> data = 'Welcome     To   BeiJing!'
>>> output = text.WhitespaceTokenizer(with_offsets=True)(data)
>>> print(output)
(array(['Welcome', 'To', 'BeiJing!'], dtype='<U8'), array([ 0, 12, 17], dtype=uint32),
array([ 7, 14, 25], dtype=uint32))
Tutorial Examples: