mindspore.dataset.text.transforms.WordpieceTokenizer

class mindspore.dataset.text.transforms.WordpieceTokenizer(vocab, suffix_indicator="##", max_bytes_per_token=100, unknown_token="[UNK]", with_offsets=False)[source]

Tokenize scalar token or 1-D tokens to 1-D subword tokens.

Parameters

  • vocab (Vocab) – A vocabulary object.

  • suffix_indicator (str, optional) – Used to show that the subword is the last part of a word (default=’##’).

  • max_bytes_per_token (int, optional) – Tokens exceeding this length will not be further split (default=100).

  • unknown_token (str, optional) – When a token cannot be found: if ‘unknown_token’ is empty string, return the token directly, else return ‘unknown_token’ (default=’[UNK]’).

  • with_offsets (bool, optional) – Whether or not output offsets of tokens (default=False).

Examples

>>> vocab_list = ["book", "cholera", "era", "favor", "##ite", "my", "is", "love", "dur", "##ing", "the"]
>>> vocab = text.Vocab.from_list(vocab_list)
>>> # If with_offsets=False, default output one column {["text", dtype=str]}
>>> tokenizer_op = text.WordpieceTokenizer(vocab=vocab, unknown_token='[UNK]',
...                                        max_bytes_per_token=100, with_offsets=False)
>>> text_file_dataset = text_file_dataset.map(operations=tokenizer_op)
>>> # If with_offsets=True, then output three columns {["token", dtype=str], ["offsets_start", dtype=uint32],
>>> #                                                   ["offsets_limit", dtype=uint32]}
>>> tokenizer_op = text.WordpieceTokenizer(vocab=vocab, unknown_token='[UNK]',
...                                       max_bytes_per_token=100, with_offsets=True)
>>> text_file_dataset = text_file_dataset.map(operations=tokenizer_op, input_columns=["text"],
...                                           output_columns=["token", "offsets_start", "offsets_limit"],
...                                           column_order=["token", "offsets_start", "offsets_limit"])