Function Differences with tf.train.linear_cosine_decay

tf.train.linear_cosine_decay

class tf.train.linear_cosine_decay(
    learning_rate,
    global_step,
    decay_steps,
    num_periods=0.5,
    alpha=0.0,
    beta=0.001,
    name=None
)

For more information, see tf.train.linear_cosine_decay.

mindspore.nn.CosineDecayLR

class mindspore.nn.CosineDecayLR(
    min_lr,
    max_lr,
    decay_steps
)(global_step)

For more information, see mindspore.nn.CosineDecayLR.

Differences

TensorFlow: The formulas are as follows: global_step = min(global_step, decay_steps) linear_decay = (decay_steps - global_step) / decay_steps cosine_decay = 0.5 \* (1 + cos(pi \* 2 \* num_periods \* global_step / decay_steps)) decayed = (alpha + linear_decay) \* cosine_decay + beta decayed_learning_rate = learning_rate \* decayed

MindSpore: The calculation logic is different from Tensorflow, the formulas are as follows: current_step = min(global_step, decay_step) decayed_learning_rate = min_lr + 0.5 \* (max_lr - min_lr) \*         (1 + cos(pi \* current_step / decay_steps))

Code Example

# The following implements CosineDecayLR with MindSpore.
import numpy as np
import tensorflow as tf
import mindspore as ms
import mindspore.nn as nn

min_lr = 0.01
max_lr = 0.1
decay_steps = 4
global_steps = ms.Tensor(2, ms.int32)
cosine_decay_lr = nn.CosineDecayLR(min_lr, max_lr, decay_steps)
result = cosine_decay_lr(global_steps)
print(result)
# Out:
# 0.055


# The following implements linear_cosine_decay with TensorFlow.
learging_rate = 0.01
global_steps = 2
output = tf.train.linear_cosine_decay(learging_rate, global_steps, decay_steps)
ss = tf.Session()
ss.run(output)
# out
# 0.0025099998