mindspore.safetensors_to_ckpt

View Source On Gitee
mindspore.safetensors_to_ckpt(file_path, save_path=None, name_map=None, file_name_regex=None, processes_num=1)[source]

Converts safetensors files into MindSpore checkpoint format and saves them to save_path. Safetensors is a reliable and portable machine learning model storage format introduced by Huggingface, used for securely storing Tensors with fast speed (zero copy).

Note

The number of multiprocess settings is related to the size of the host, and it is not recommended to set it too large, otherwise it may cause freezing.

Parameters

  • file_path (str) – Path to the directory containing safetensors files or a single safetensors file (.safetensors).

  • save_path (str, optional) – Directory path where checkpoint files will be saved. Defaults: None.

  • name_map (dict, optional) – Dictionary mapping original parameter names to new names. Defaults: None.

  • file_name_regex (str, optional) – Regular expression used to match the file that needs to be converted. Defaults: None.

  • processes_num (int, optional) – Number of processes to use for parallel processing. Defaults: 1.

Raises

ValueError – If the input path is invalid, the save_path is not a directory, or the file_path does not end with '.safetensors'.

Supported Platforms:

Ascend GPU CPU

Examples

>>> import mindspore as ms
>>> ms.safetensors_to_ckpt("./safetensors_save_path")
>>> ms.safetensors_to_ckpt("./safetensors_save_path/rank0/checkpoint_0.safetensors")
>>> ms.safetensors_to_ckpt("./safetensors_save_path/rank0/checkpoint_0.safetensors", "./new_path/")
>>> namemap = {"lin.weight":"new_name"}
>>> ms.safetensors_to_ckpt("./safetensors_save_path/rank0/checkpoint_0.safetensors", "./new_path/", namemap)