mindspore.merge_sliced_parameter

mindspore.merge_sliced_parameter(sliced_parameters, strategy=None)[source]

Merge parameter slices into one parameter. Used in the case of distributed inference. For details of merge_sliced_parameter, please check: Enabling Graph-Accounting Convergence.

Parameters

  • sliced_parameters (list[Parameter]) – Parameter slices in order of rank_id.

  • strategy (Optional[dict]) – Parameter slice strategy, whose key is parameter name and value is slice strategy of this parameter. If strategy is None, just merge parameter slices in 0 axis order. Default: None.

Returns

Parameter, the merged parameter which has the whole data.

Raises

  • ValueError – Failed to merge.

  • TypeError – The sliced_parameters is incorrect or strategy is not dict.

  • KeyError – The parameter name is not in keys of strategy.

Examples

>>> import numpy as np
>>> from mindspore import Tensor, merge_sliced_parameter, Parameter
>>>
>>> sliced_parameters = [
...                      Parameter(Tensor(np.array([0.00023915, 0.00013939, -0.00098059])),
...                                "network.embedding_table"),
...                      Parameter(Tensor(np.array([0.00015815, 0.00015458, -0.00012125])),
...                                "network.embedding_table"),
...                      Parameter(Tensor(np.array([0.00042165, 0.00029692, -0.00007941])),
...                                "network.embedding_table"),
...                      Parameter(Tensor(np.array([0.00084451, 0.00089960, -0.00010431])),
...                                "network.embedding_table")]
>>> merged_parameter = merge_sliced_parameter(sliced_parameters)
>>> print(merged_parameter)
Parameter (name=network.embedding_table, shape=(12,), dtype=Float64, requires_grad=True)