mindspore.dataset.text.NormalizeUTF8

class mindspore.dataset.text.NormalizeUTF8(normalize_form=NormalizeForm.NFKC)[源代码]

对UTF-8编码的字符串进行规范化处理。

说明

Windows平台尚不支持 NormalizeUTF8

参数:
  • normalize_form (NormalizeForm, 可选) - 想要使用的规范化模式。可选值详见 NormalizeForm 。 默认值: NormalizeForm.NFKC

异常:
支持平台:

CPU

样例:

>>> import mindspore.dataset as ds
>>> import mindspore.dataset.text as text
>>> from mindspore.dataset.text import NormalizeForm
>>>
>>> # Use the transform in dataset pipeline mode
>>> numpy_slices_dataset = ds.NumpySlicesDataset(data=["ṩ", "ḍ̇", "q̇", "fi", "2⁵", "ẛ"],
...                                              column_names=["text"], shuffle=False)
>>> normalize_op = text.NormalizeUTF8(normalize_form=NormalizeForm.NFC)
>>> numpy_slices_dataset = numpy_slices_dataset.map(operations=normalize_op)
>>> for item in numpy_slices_dataset.create_dict_iterator(num_epochs=1, output_numpy=True):
...     print(item["text"])
...     break

>>>
>>> # Use the transform in eager mode
>>> data = ["ṩ", "ḍ̇", "q̇", "fi", "2⁵", "ẛ"]
>>> output = text.NormalizeUTF8(NormalizeForm.NFKC)(data)
>>> print(output)
['ṩ' 'ḍ̇' 'q̇' 'fi' '25' 'ṡ']
教程样例: