mindspore.ckpt_to_safetensors

mindspore.ckpt_to_safetensors(file_path, save_path=None, name_map=None, file_name_regex=None, processes_num=1)[source]

Converts MindSpore checkpoint files into safetensors format and saves them to save_path. Safetensors is a reliable and portable machine learning model storage format introduced by Huggingface, used for securely storing Tensors with fast speed (zero copy).

Note

The number of multiprocess settings is related to the size of the host, and it is not recommended to set it too large, otherwise it may cause freezing. The safetensors format does not support the enc verification function. If ckpt is enabled to save enc verification, an error will be generated when performing the conversion. The safetensors format currently does not support crc verification function. If ckpt contains crc verification information, the crc verification information will be lost after conversion to safetensors.

Parameters

file_path (str) – Path to the directory containing checkpoint files or a single checkpoint file (.ckpt).
save_path (str, optional) – Directory path where safetensors files will be saved. Defaults: None.
name_map (dict, optional) – Dictionary mapping original parameter names to new names. Defaults: None.
file_name_regex (str, optional) – Regular expression used to match the file that needs to be converted. Defaults: None.
processes_num (int, optional) – Number of processes to use for parallel processing. Defaults: 1.

Raises

ValueError – If the input path is invalid or the save_path is not a directory, or the file_path does not end with '.ckpt'.

Supported Platforms:: Ascend GPU CPU

Examples

>>> import mindspore as ms
>>> ms.ckpt_to_safetensors("./ckpt_save_path")
>>> ms.ckpt_to_safetensors("./ckpt_save_path/rank0/checkpoint_0.ckpt")
>>> ms.ckpt_to_safetensors(file_path="./ckpt_save_path/rank0/checkpoint_0.ckpt", save_path="./new_path/")
>>> namemap = {"lin.weight":"new_name"}
>>> ms.ckpt_to_safetensors("./ckpt_save_path/rank0/checkpoint_0.ckpt", "./new_path/", namemap)