mindspore.dataset.text.CharNGram

class mindspore.dataset.text.CharNGram

CharNGram object that is used to map tokens into pre-trained vectors.

classmethod from_file(file_path, max_vectors=None)

Build a CharNGram vector from a file.

Parameters

• file_path (str) – Path of the file that contains the CharNGram vectors.

• max_vectors (int, optional) – This can be used to limit the number of pre-trained vectors loaded. Most pre-trained vector sets are sorted in the descending order of word frequency. Thus, in situations where the entire set doesn’t fit in memory, or is not needed for another reason, passing max_vectors can limit the size of the loaded set (default=None, no limit).

Examples

>>> char_n_gram = text.CharNGram.from_file("/path/to/char_n_gram/file", max_vectors=None)