比较与torchtext.data.utils.ngrams_iterator的功能差异

torchtext.data.utils.ngrams_iterator

torchtext.data.utils.ngrams_iterator(
    token_list,
    ngrams
)

更多内容详见torchtext.data.utils.ngrams_iterator。

mindspore.dataset.text.transforms.Ngram

class mindspore.dataset.text.transforms.Ngram(
    n,
    left_pad=("", 0),
    right_pad=("", 0),
    separator=" "
)

更多内容详见mindspore.dataset.text.transforms.Ngram。

使用方式

PyTorch：返回一个迭代器，该迭代器生成给定的标记和ngrams。

MindSpore：TensorOp从一维字符串张量生成n-gram。

代码示例

from mindspore.dataset import text
from torchtext.data.utils import ngrams_iterator

# In MindSpore, output numpy.ndarray type n-gram.

ngram_op = text.Ngram(3, separator="-")
output = ngram_op(["WildRose Country", "Canada's Ocean Playground", "Land of Living Skies"])
print(output)
# Out:
# ["WildRose Country-Canada's Ocean Playground-Land of Living Skies"]

# In torch, return an iterator that yields the given tokens and their ngrams.
token_list = ['here', 'we', 'are']
print(list(ngrams_iterator(token_list, 2)))
# Out:
# ['here', 'we', 'are', 'here we', 'we are']