Class AddToken

• Defined in File text.h

Inheritance Relationships

Base Type

• public mindspore::dataset::TensorTransform (Class TensorTransform)

Class Documentation

class AddToken : public mindspore::dataset::TensorTransform

Add token to beginning or end of sequence.

Public Functions

explicit AddToken(const std::string &token, bool begin = true)

Constructor.

Parameters

• token – [in] The token to be added.

• begin – [in] Whether to insert token at start or end of sequence. Default: true.

Example

/* Define operations */
auto add_token_op = text::AddToken(token='TOKEN', begin=True);

/* dataset is an instance of Dataset object */
dataset = dataset->Map({add_token_op},   // operations
                       {"text"});       // input columns

~AddToken() override = default

Destructor.