mindspore.dataset.text.WhitespaceTokenizer
- class mindspore.dataset.text.WhitespaceTokenizer(with_offsets=False)[source]
Tokenize a scalar tensor of UTF-8 string on ICU4C defined whitespaces, such as: ‘ ‘, ‘\t’, ‘\r’, ‘\n’.
Note
WhitespaceTokenizer is not supported on Windows platform yet.
- Parameters
with_offsets (bool, optional) – Whether or not output offsets of tokens (default=False).
- Raises
TypeError – If with_offsets is not of type bool.
- Supported Platforms:
CPU
Examples
>>> # If with_offsets=False, default output one column {["text", dtype=str]} >>> tokenizer_op = text.WhitespaceTokenizer(with_offsets=False) >>> text_file_dataset = text_file_dataset.map(operations=tokenizer_op) >>> # If with_offsets=True, then output three columns {["token", dtype=str], >>> # ["offsets_start", dtype=uint32], >>> # ["offsets_limit", dtype=uint32]} >>> tokenizer_op = text.WhitespaceTokenizer(with_offsets=True) >>> text_file_dataset = text_file_dataset.map(operations=tokenizer_op, input_columns=["text"], ... output_columns=["token", "offsets_start", "offsets_limit"], ... column_order=["token", "offsets_start", "offsets_limit"])