mindspore.mint.nn.GLU

class mindspore.mint.nn.GLU(dim=- 1)

Computes GLU (Gated Linear Unit activation function) of the input tensor.

$$GLU(a,b)=a\otimes \sigma (b)$$

where $a$ is the first half of the input Tensor after input is split and $b$ is the second half.

Here $\sigma$ is the sigmoid function, and $\otimes$ is the Hadamard product. See Language Modeling with Gated Convluational Networks .

Parameters

dim (int, optional) – The dimension to split the input input. The value range is [-r, r) where r is the number of dimensions of input. Default: -1 , the last dimension in input.

Inputs:

• input (Tensor) - Tensor to be calculated. Dtype is floating point and the shape is $({\ast }_1,N,{\ast }_2)$ where * means, any number of additional dimensions. $N$ is required to be an even number, where $N$ is the size of input on the dimension selected by dim.

Outputs:

Tensor, the same dtype as the input, with the shape $({\ast }_1,M,{\ast }_2)$ where $M=N/2$.

Raises

• TypeError – If input is not a Tensor or dim is not an int.

• IndexError – If the value of dim is out of the range of [-r, r), where r is the number of dimensions of input.

• RuntimeError – If dtype of input is not supported.

• RuntimeError – If the length of input in the dimension selected by dim is not even.

Supported Platforms:

Ascend

Examples

>>> from mindspore import mint, Tensor
>>> glu = mint.nn.GLU()
>>> input = Tensor([[0.1, 0.2, 0.3, 0.4], [0.5, 0.6, 0.7, 0.8]])
>>> output = glu(input)
>>> print(output)
[[0.05744425 0.11973753]
 [0.33409387 0.41398472]]