mindspore.dataset.text.AddToken

class mindspore.dataset.text.AddToken(token, begin=True)[source]

Add token to beginning or end of sequence.

Parameters

  • token (str) – The token to be added.

  • begin (bool, optional) – Choose the position where the token is inserted. If True, the token will be inserted at the beginning of the sequence. Otherwise, it will be inserted at the end of the sequence. Default: True.

Raises

  • TypeError – If token is not of type string.

  • TypeError – If begin is not of type bool.

Supported Platforms:

CPU

Examples

>>> import mindspore.dataset as ds
>>> import mindspore.dataset.text as text
>>>
>>> dataset = ds.NumpySlicesDataset(data={"text": [['a', 'b', 'c', 'd', 'e']]})
>>> # Data before
>>> # |           text            |
>>> # +---------------------------+
>>> # | ['a', 'b', 'c', 'd', 'e'] |
>>> # +---------------------------+
>>> add_token_op = text.AddToken(token='TOKEN', begin=True)
>>> dataset = dataset.map(operations=add_token_op)
>>> # Data after
>>> # |           text            |
>>> # +---------------------------+
>>> # | ['TOKEN', 'a', 'b', 'c', 'd', 'e'] |
>>> # +---------------------------+
Tutorial Examples: