mindspore.ops.gelu

mindspore.ops.gelu(input, approximate='none')

Gaussian Error Linear Units activation function.

GeLU is described in the paper Gaussian Error Linear Units (GELUs). And also please refer to BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.

When approximate argument is none, GeLU is defined as follows:

$$GELU(x_i)=x_i\ast P(X<x_i),$$

where $P$ is the cumulative distribution function of the standard Gaussian distribution, $x_i$ is the input element.

When approximate argument is tanh, GeLU is estimated with:

$$GELU(x_i)=0.5\ast x_i\ast (1+\tanh (\sqrt{(}2/\pi )\ast (x_i+0.044715\ast x_i^3)))$$

GELU Activation Function Graph:

Parameters

• input (Tensor) – The input of the activation function GeLU, the data type is float16, float32 or float64.

• approximate (str, optional) – the gelu approximation algorithm to use. Acceptable vaslues are 'none' and 'tanh' . Default: 'none' .

Returns

Tensor, with the same type and shape as input.

Raises

• TypeError – If input is not a Tensor.

• TypeError – If dtype of input is not bfloat16, float16, float32 or float64.

• ValueError – If approximate value is neither none nor tanh.

Supported Platforms:

Ascend GPU CPU

Examples

>>> import mindspore
>>> from mindspore import Tensor, ops
>>> x = Tensor([1.0, 2.0, 3.0], mindspore.float32)
>>> result = ops.gelu(x, approximate='none')
>>> print(result)
[0.8413447 1.9544997 2.9959505]