Document feedback

Question document fragment

When a question document fragment contains a formula, it is displayed as a space.

Submission type

issue

It's a little complicated...

I'd like to ask someone.

Please select the submission type

Problem type

Specifications and Common Mistakes

- Specifications and Common Mistakes:

- Misspellings or punctuation mistakes,incorrect formulas, abnormal display.

- Incorrect links, empty cells, or wrong formats.

- Chinese characters in English context.

- Minor inconsistencies between the UI and descriptions.

- Low writing fluency that does not affect understanding.

- Incorrect version numbers, including software package names and version numbers on the UI.

Usability

- Usability:

- Incorrect or missing key steps.

- Missing main function descriptions, keyword explanation, necessary prerequisites, or precautions.

- Ambiguous descriptions, unclear reference, or contradictory context.

- Unclear logic, such as missing classifications, items, and steps.

Correctness

- Correctness:

- Technical principles, function descriptions, supported platforms, parameter types, or exceptions inconsistent with that of software implementation.

- Incorrect schematic or architecture diagrams.

- Incorrect commands or command parameters.

- Incorrect code.

- Commands inconsistent with the functions.

- Wrong screenshots.

- Sample code running error, or running results inconsistent with the expectation.

Risk Warnings

- Risk Warnings:

- Lack of risk warnings for operations that may damage the system or important data.

Content Compliance

- Content Compliance:

- Contents that may violate applicable laws and regulations or geo-cultural context-sensitive words and expressions.

- Copyright infringement.

Please select the type of question

Problem description

Describe the bug so that we can quickly locate the problem.

Submit

Document feedback

mindspore.nn.LARS

class mindspore.nn.LARS(optimizer, epsilon=1e-05, coefficient=0.001, use_clip=False, lars_filter=lambda x: ...)

Implements the LARS algorithm.

LARS is an optimization algorithm employing a large batch optimization technique. Refer to paper LARGE BATCH TRAINING OF CONVOLUTIONAL NETWORKS.

The updating formulas are as follows,

$$\begin{array}{r}\begin{array}{l}\\ & \\ & \\ & \mathbf{\text{Parameters}}:\text{base learning rate }{\gamma }_0,\text{ momentum m},\text{ weight decay }\lambda ,\\ & \text{ LARS coefficient }\eta ,\text{ number of steps }T\\ & \mathbf{\text{Init}}:\text{ t=0, v=0, init weight }{w}_0^l\text{ for each layer }l\\ & \\ & \\ & \mathbf{\text{while}}\text{ t<T for each layer }l\mathbf{\text{ do}}\\ & g_t^l\leftarrow \mathrm{\nabla }L\left(w_t^l\right)\\ & {\gamma }_t\leftarrow {\gamma }_0\ast {(1-\frac{t}{T})}^2\\ & {\gamma }^l\leftarrow \eta \ast \frac{\Vert w_t^l\Vert }{\Vert g_t^l\Vert +\lambda \Vert w_t^l\Vert }\text{(compute the local LR }{\gamma }^{l)}\\ & v_{t+1}^l\leftarrow m{v}_t^l+{\gamma }_{t+1}\ast {\gamma }^l\ast (g_t^l+\lambda w_t^l)\\ & w_{t+1}^l\leftarrow w_t^l-v_{t+1}^l\\ & \mathbf{\text{ end while }}\\ & \\ & \end{array}\end{array}$$

$w$ represents the network parameters, $g$ represents gradients, $t$ represents the current step, $\lambda$ represents weight_decay in optimizer, $\gamma$ represents learning_rate in optimizer, $\eta$ represents coefficient.

Parameters

• optimizer (Optimizer) – MindSpore optimizer for which to wrap and modify gradients.

• epsilon (float) – Term added to the denominator to improve numerical stability. Default: 1e-05.

• coefficient (float) – Trust coefficient for calculating the local learning rate. Default: 0.001.

• use_clip (bool) – Whether to use clip operation for calculating the local learning rate. Default: False.

• lars_filter (Function) – A function to determine which of the network parameters to use LARS algorithm. Default: lambda x: ‘LayerNorm’ not in x.name and ‘bias’ not in x.name.

Inputs:

• gradients (tuple[Tensor]) - The gradients of params in the optimizer, the shape is the as same as the params in the optimizer.

Outputs:

Union[Tensor[bool], tuple[Parameter]], it depends on the output of optimizer.

Supported Platforms:

Ascend

Examples

>>> import mindspore as ms
>>> from mindspore import nn
>>>
>>> # Define the network structure of LeNet5. Refer to
>>> # https://gitee.com/mindspore/docs/blob/r2.0/docs/mindspore/code/lenet.py
>>> net = LeNet5()
>>> loss = nn.SoftmaxCrossEntropyWithLogits()
>>> opt = nn.Momentum(net.trainable_params(), 0.1, 0.9)
>>> opt_lars = nn.LARS(opt, epsilon=1e-08, coefficient=0.02)
>>> model = ms.Model(net, loss_fn=loss, optimizer=opt_lars, metrics=None)