mindspore.nn.thor
- mindspore.nn.thor(net, learning_rate, damping, momentum, weight_decay=0.0, loss_scale=1.0, batch_size=32, use_nesterov=False, decay_filter=<lambda x: x.name not in []>, split_indices=None, enable_clip_grad=False, frequency=100)[source]
Updates gradients by second-order algorithm–THOR.
Trace-based Hardware-driven layer-ORiented Natural Gradient Descent Computation (THOR) algorithm is proposed in:
THOR: Trace-based Hardware-driven layer-ORiented Natural Gradient Descent Computation
The updating formulas are as follows,
Note
When separating parameter groups, the weight decay in each group will be applied on the parameters if the weight decay is positive. When not separating parameter groups, the weight_decay in the API will be applied on the parameters without ‘beta’ or ‘gamma’ in their names if weight_decay is positive.
When separating parameter groups, if you want to centralize the gradient, set grad_centralization to True, but the gradient centralization can only be applied to the parameters of the convolution layer. If the parameters of the non convolution layer are set to True, an error will be reported.
To improve parameter groups performance, the customized order of parameters can be supported.
- Parameters
net (Cell) – The training network.
learning_rate (Tensor) – A value for the learning rate.
damping (Tensor) – A value for the damping.
momentum (float) – Hyper-parameter of type float, means momentum for the moving average. It must be at least 0.0.
weight_decay (int, float) – Weight decay (L2 penalty). It must be equal to or greater than 0.0. Default: 0.0.
loss_scale (float) – A value for the loss scale. It must be greater than 0.0. In general, use the default value. Default: 1.0.
batch_size (int) – The size of a batch. Default: 32
use_nesterov (bool) – Enable Nesterov momentum. Default: False.
decay_filter (function) – A function to determine which layers the weight decay applied to. And it only works when the weight_decay > 0. Default: lambda x: x.name not in []
split_indices (list) – Set allreduce fusion strategy by A/G layer indices . Only works when distributed computing. ResNet50 as an example, there are 54 layers of A/G respectively, when split_indices is set to [26, 53], it means A/G is divided into two groups to allreduce, one is 0~26 layer, and the other is 27~53. Default: None
enable_clip_grad (bool) – Whether to clip the gradients. Default: False
frequency (int) – The update interval of A/G and $A^{-1}/G^{-1}$. When frequency equals N (N is greater than 1), A/G and $A^{-1}/G^{-1}$ will be updated every N steps, and other steps will use the stale A/G and $A^{-1}/G^{-1}$ to update weights. Default: 100.
- Inputs:
gradients (tuple[Tensor]) - The gradients of params, the shape is the same as params.
- Outputs:
tuple[bool], all elements are True.
- Raises
TypeError – If learning_rate is not Tensor.
TypeError – If loss_scale,`momentum` or frequency is not a float.
TypeError – If weight_decay is neither float nor int.
TypeError – If use_nesterov is not a bool.
ValueError – If loss_scale is less than or equal to 0.
ValueError – If weight_decay or momentum is less than 0.
ValueError – If frequency is not int.
ValueError – If frequency is less than 2.
- Supported Platforms:
AscendGPU
Examples
>>> net = Net() >>> optim = thor(net, lr=Tensor(1e-3), damping=Tensor(1e-3), momentum=0.9) >>> loss = nn.SoftmaxCrossEntropyWithLogits() >>> model = ConvertModelUtils().convert_to_thor_model(model=model, network=net, loss_fn=loss, optimizer=opt, ... loss_scale_manager=loss_scale, metrics={'acc'}, amp_level="O2", keep_batchnorm_fp32=False) >>> model.train(config.epoch_size, dataset, callbacks=cb, sink_size=100, dataset_sink_mode=True)
