mindspore.dataset.text.NormalizeUTF8
- class mindspore.dataset.text.NormalizeUTF8(normalize_form=NormalizeForm.NFKC)[源代码]
对UTF-8编码的字符串进行规范化处理。
说明
Windows平台尚不支持 NormalizeUTF8 。
- 参数:
normalize_form (
NormalizeForm
, 可选) - 想要使用的规范化模式。可选值详见NormalizeForm
。 默认值:NormalizeForm.NFKC
。
- 异常:
TypeError - 当 normalize_form 不为
NormalizeForm
类型。
- 支持平台:
CPU
样例:
>>> import mindspore.dataset as ds >>> import mindspore.dataset.text as text >>> from mindspore.dataset.text import NormalizeForm >>> >>> # Use the transform in dataset pipeline mode >>> numpy_slices_dataset = ds.NumpySlicesDataset(data=["ṩ", "ḍ̇", "q̇", "fi", "2⁵", "ẛ"], ... column_names=["text"], shuffle=False) >>> normalize_op = text.NormalizeUTF8(normalize_form=NormalizeForm.NFC) >>> numpy_slices_dataset = numpy_slices_dataset.map(operations=normalize_op) >>> for item in numpy_slices_dataset.create_dict_iterator(num_epochs=1, output_numpy=True): ... print(item["text"]) ... break ṩ >>> >>> # Use the transform in eager mode >>> data = ["ṩ", "ḍ̇", "q̇", "fi", "2⁵", "ẛ"] >>> output = text.NormalizeUTF8(NormalizeForm.NFKC)(data) >>> print(output) ['ṩ' 'ḍ̇' 'q̇' 'fi' '25' 'ṡ']
- 教程样例: