mindspore.nn.dynamic_lr

Dynamic Learning Rate

mindspore.nn.dynamic_lr.cosine_decay_lr(min_lr, max_lr, total_step, step_per_epoch, decay_epoch)

Calculate learning rate base on cosine decay function.

For the i-th step, the formula of computing decayed_learning_rate[i] is:

$$decayed\mathrm{\_}learning\mathrm{\_}rate[i]=min\mathrm{\_}learning\mathrm{\_}rate+0.5\ast (max\mathrm{\_}learning\mathrm{\_}rate-min\mathrm{\_}learning\mathrm{\_}rate)\ast (1+cos(\frac{current\mathrm{\_}epoch}{decay\mathrm{\_}epoch}\pi ))$$

Where $current\mathrm{\_}epoch=floor(\frac{i}{step\mathrm{\_}per\mathrm{\_}epoch})$.

Parameters

• min_lr (float) – The minimum value of learning rate.

• max_lr (float) – The maximum value of learning rate.

• total_step (int) – The total number of steps.

• step_per_epoch (int) – The number of steps in per epoch.

• decay_epoch (int) – A value used to calculate decayed learning rate.

Returns

list[float]. The size of list is total_step.

Examples

>>> min_lr = 0.01
>>> max_lr = 0.1
>>> total_step = 6
>>> step_per_epoch = 2
>>> decay_epoch = 2
>>> cosine_decay_lr(min_lr, max_lr, total_step, step_per_epoch, decay_epoch)
[0.1, 0.1, 0.05500000000000001, 0.05500000000000001, 0.01, 0.01]

mindspore.nn.dynamic_lr.exponential_decay_lr(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, is_stair=False)

Calculate learning rate base on exponential decay function.

For the i-th step, the formula of computing decayed_learning_rate[i] is:

$$decayed\mathrm{\_}learning\mathrm{\_}rate[i]=learning\mathrm{\_}rate\ast decay\mathrm{\_}rat{e}^{\frac{current\mathrm{\_}epoch}{decay\mathrm{\_}epoch}}$$

Where $current\mathrm{\_}epoch=floor(\frac{i}{step\mathrm{\_}per\mathrm{\_}epoch})$.

Parameters

• learning_rate (float) – The initial value of learning rate.

• decay_rate (float) – The decay rate.

• total_step (int) – The total number of steps.

• step_per_epoch (int) – The number of steps in per epoch.

• decay_epoch (int) – A value used to calculate decayed learning rate.

• is_stair (bool) – If true, learning rate is decayed once every decay_epoch times. Default: False.

Returns

list[float]. The size of list is total_step.

Examples

>>> learning_rate = 0.1
>>> decay_rate = 0.9
>>> total_step = 6
>>> step_per_epoch = 2
>>> decay_epoch = 1
>>> exponential_decay_lr(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch)
[0.1, 0.1, 0.09000000000000001, 0.09000000000000001, 0.08100000000000002, 0.08100000000000002]

mindspore.nn.dynamic_lr.inverse_decay_lr(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, is_stair=False)

Calculate learning rate base on inverse-time decay function.

For the i-th step, the formula of computing decayed_learning_rate[i] is:

$$decayed\mathrm{\_}learning\mathrm{\_}rate[i]=learning\mathrm{\_}rate/(1+decay\mathrm{\_}rate\ast current\mathrm{\_}epoch/decay\mathrm{\_}epoch)$$

Where $current\mathrm{\_}epoch=floor(\frac{i}{step\mathrm{\_}per\mathrm{\_}epoch})$.

Parameters

• learning_rate (float) – The initial value of learning rate.

• decay_rate (float) – The decay rate.

• total_step (int) – The total number of steps.

• step_per_epoch (int) – The number of steps in per epoch.

• decay_epoch (int) – A value used to calculate decayed learning rate.

• is_stair (bool) – If true, learning rate is decayed once every decay_epoch times. Default: False.

Returns

list[float]. The size of list is total_step.

Examples

>>> learning_rate = 0.1
>>> decay_rate = 0.5
>>> total_step = 6
>>> step_per_epoch = 1
>>> decay_epoch = 1
>>> inverse_decay_lr(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, True)
[0.1, 0.06666666666666667, 0.05, 0.04, 0.03333333333333333, 0.028571428571428574]

mindspore.nn.dynamic_lr.natural_exp_decay_lr(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, is_stair=False)

Calculate learning rate base on natural exponential decay function.

For the i-th step, the formula of computing decayed_learning_rate[i] is:

$$decayed\mathrm{\_}learning\mathrm{\_}rate[i]=learning\mathrm{\_}rate\ast e^{-decay\mathrm{\_}rate\ast current\mathrm{\_}epoch}$$

Where $current\mathrm{\_}epoch=floor(\frac{i}{step\mathrm{\_}per\mathrm{\_}epoch})$.

Parameters

• learning_rate (float) – The initial value of learning rate.

• decay_rate (float) – The decay rate.

• total_step (int) – The total number of steps.

• step_per_epoch (int) – The number of steps in per epoch.

• decay_epoch (int) – A value used to calculate decayed learning rate.

• is_stair (bool) – If true, learning rate is decayed once every decay_epoch times. Default: False.

Returns

list[float]. The size of list is total_step.

Examples

>>> learning_rate = 0.1
>>> decay_rate = 0.9
>>> total_step = 6
>>> step_per_epoch = 2
>>> decay_epoch = 2
>>> natural_exp_decay_lr(learning_rate, decay_rate, total_step, step_per_epoch, decay_epoch, True)
[0.1, 0.1, 0.1, 0.1, 0.016529888822158657, 0.016529888822158657]

mindspore.nn.dynamic_lr.piecewise_constant_lr(milestone, learning_rates)

Get piecewise constant learning rate.

Calculate learning rate by given milestone and learning_rates. Let the value of milestone be $(M_1,M_2,...,M_N)$ and the value of learning_rates be $(x_1,x_2,...,x_N)$. N is the length of milestone. Let the output learning rate be y.

$$y[i]=x_t,fori\in [M_{t-1},M_t)$$

Parameters

• milestone (Union[list[int], tuple[int]]) – A list of milestone. This list is a monotone increasing list. Every element is a milestone step, and must be greater than 0.

• learning_rates (Union[list[float], tuple[float]]) – A list of learning rates.

Returns

list[float]. The size of list is $M_N$.

Examples

>>> milestone = [2, 5, 10]
>>> learning_rates = [0.1, 0.05, 0.01]
>>> piecewise_constant_lr(milestone, learning_rates)
[0.1, 0.1, 0.05, 0.05, 0.05, 0.01, 0.01, 0.01, 0.01, 0.01]

mindspore.nn.dynamic_lr.polynomial_decay_lr(learning_rate, end_learning_rate, total_step, step_per_epoch, decay_epoch, power, update_decay_epoch=False)

Calculate learning rate base on polynomial decay function.

For the i-th step, the formula of computing decayed_learning_rate[i] is:

$$decayed\mathrm{\_}learning\mathrm{\_}rate[i]=(learning\mathrm{\_}rate-end\mathrm{\_}learning\mathrm{\_}rate)\ast (1-tmp\mathrm{\_}epoch/tmp\mathrm{\_}decay\mathrm{\_}epoch{)}^{power}+end\mathrm{\_}learning\mathrm{\_}rate$$

Where:

$$tmp\mathrm{\_}epoch=min(current\mathrm{\_}epoch,decay\mathrm{\_}epoch)$$

$$current\mathrm{\_}epoch=floor(\frac{i}{step\mathrm{\_}per\mathrm{\_}epoch})$$

$$tmp\mathrm{\_}decay\mathrm{\_}epoch=decay\mathrm{\_}epoch$$

If update_decay_epoch is true, update the value of tmp_decay_epoch every epoch. The formula is:

$$tmp\mathrm{\_}decay\mathrm{\_}epoch=decay\mathrm{\_}epoch\ast ceil(current\mathrm{\_}epoch/decay\mathrm{\_}epoch)$$

Parameters

• learning_rate (float) – The initial value of learning rate.

• end_learning_rate (float) – The end value of learning rate.

• total_step (int) – The total number of steps.

• step_per_epoch (int) – The number of steps in per epoch.

• decay_epoch (int) – A value used to calculate decayed learning rate.

• power (float) – A value used to calculate decayed learning rate. This parameter must be greater than 0.

• update_decay_epoch (bool) – If true, update decay_epoch. Default: False.

Returns

list[float]. The size of list is total_step.

Examples

>>> learning_rate = 0.1
>>> end_learning_rate = 0.01
>>> total_step = 6
>>> step_per_epoch = 2
>>> decay_epoch = 2
>>> power = 0.5
>>> polynomial_decay_lr(learning_rate, end_learning_rate, total_step, step_per_epoch, decay_epoch, power)
[0.1, 0.1, 0.07363961030678928, 0.07363961030678928, 0.01, 0.01]

mindspore.nn.dynamic_lr.warmup_lr(learning_rate, total_step, step_per_epoch, warmup_epoch)

Get learning rate warming up.

For the i-th step, the formula of computing warmup_learning_rate[i] is:

$$warmup\mathrm{\_}learning\mathrm{\_}rate[i]=learning\mathrm{\_}rate\ast tmp\mathrm{\_}epoch/tmp\mathrm{\_}warmup\mathrm{\_}epoch$$

Where $tmp\mathrm{\_}epoch=min(current\mathrm{\_}epoch,warmup\mathrm{\_}epoch),current\mathrm{\_}epoch=floor(\frac{i}{step\mathrm{\_}per\mathrm{\_}epoch})$

Parameters

• learning_rate (float) – The initial value of learning rate.

• warmup_steps (int) – The warm up steps of learning rate.

Inputs:

Tensor. The current step number.

Returns

Tensor. The learning rate value for the current step.

Examples

>>> learning_rate = 0.1
>>> total_step = 6
>>> step_per_epoch = 2
>>> warmup_epoch = 2
>>> warmup_lr(learning_rate, total_step, step_per_epoch, warmup_epoch)
[0.0, 0.0, 0.05, 0.05, 0.1, 0.1]