Function Differences with tf.keras.layers.LayerNormalization
tf.keras.layers.LayerNormalization
class tf.keras.layers.LayerNormalization(
axis=-1,
epsilon=0.001,
center=True,
scale=True,
beta_initializer='zeros',
gamma_initializer='ones',
beta_regularizer=None,
gamma_regularizer=None,
beta_constraint=None,
gamma_constraint=None,
**kwargs
)(x) -> Tensor
For more information, see tf.keras.layers.LayerNormalization.
mindspore.nn.LayerNorm
class mindspore.nn.LayerNorm(
normalized_shape,
begin_norm_axis=-1,
begin_params_axis=-1,
gamma_init='ones',
beta_init='zeros',
epsilon=1e-7
)(x) -> Tensor
For more information, see mindspore.nn.LayerNorm.
Differences
TensorFlow: Apply Layer Normalization to the mini-batch input, where the parameters center
and scale
control whether to use beta and gamma, beta_regularizer
and gamma_regularizer
are used to control whether to use optional regularizers for beta and gamma, and the parameters beta_constraint
and gamma_constraint
are used to control whether to use optional constraints for beta and gamma.
MindSpore: When all the parameters in this API of TensorFlow are default, MindSpore API basically implements the same function as TensorFlow. However, the parameters center
and scale
do not exist in MindSpore, so the function of ignoring beta and gamma cannot be implemented; the parameters beta_regularizer
, gamma_regularizer
, beta_constraint
, and gamma_constraint
do not exist, so the corresponding functions are not supported. Also MindSpore adds the parameter begin_params_axis
to control the dimensionality of the first parameter (beta, gamma) and the parameter normalized_shape
to control the specific dimensionality of the mean and standard deviation calculation.
Categories |
Subcategories |
TensorFlow |
MindSpore |
Differences |
---|---|---|---|---|
Parameters |
Parameter 1 |
axis |
begin_norm_axis |
Same function, different parameter name |
Parameter 2 |
epsilon |
epsilon |
Same function, same parameter name, different default value |
|
Parameter 3 |
center |
- |
This parameter is used in TensorFlow to control whether to use the beta offset. MindSpore does not have this parameter |
|
Parameter 4 |
scale |
- |
This parameter is used in TensorFlow to control whether gamma is used. MindSpore does not have this parameter |
|
Parameter 5 |
beta_initializer |
beta_init |
Same function, different parameter names |
|
Parameter 6 |
gamma_initializer |
gamma_init |
Same function, different parameter names |
|
Parameter 7 |
beta_regularizer |
- |
This parameter is used in TensorFlow to control whether an optional regularizer with beta weights is used. MindSpore does not have this parameter |
|
Parameter 8 |
gamma_regularizer |
- |
This parameter is used in TensorFlow to control whether the optional regularizer with gamma weights is used. MindSpore does not have this parameter |
|
Parameter 9 |
beta_constraint |
- |
This parameter is used in TensorFlow to control whether the optional constraint of beta weights is used. MindSpore does not have this parameter |
|
Parameter 10 |
gamma_constraint |
- |
This parameter is used in TensorFlow to control whether the optional constraint of gamma weights is used. MindSpore does not have this parameter |
|
Parameter 11 |
**kwargs |
- |
Not involved |
|
Parameter 12 |
- |
normalized_shape |
This parameter is not available in TensorFlow, and this parameter in MindSpore controls the specific dimension of the mean and standard deviation calculation |
|
Parameter 13 |
- |
begin_params_axis |
No such parameter in TensorFlow. This parameter in MindSpore determines the dimension of the first parameter (beta, gamma) and is used to broadcast the built-in parameters scale and centering |
|
Input |
Single input |
x |
x |
- |
Code Example
When all parameters in this API of TensorFlow are default, the two APIs achieve the same function and have the same usage.
# TensorFlow
import tensorflow as tf
inputs = tf.ones([20, 5, 10, 10])
layer = tf.keras.layers.LayerNormalization(axis=-1)
output = layer(inputs)
print(output.shape)
# (20, 5, 10, 10)
# MindSpore
import mindspore
from mindspore import Tensor
import mindspore.numpy as np
import mindspore.nn as nn
x = Tensor(np.ones([20, 5, 10, 10]), mindspore.float32)
shape1 = x.shape[1:]
m = nn.LayerNorm(shape1, begin_norm_axis=1, begin_params_axis=1)
output = m(x).shape
print(output)
# (20, 5, 10, 10)