mojolearn.layer_norm_forward#

mojolearn.layer_norm_forward(x, normalized_shape=None, weight=None, bias=None, eps=1e-05, numeric_mode=None)[source]#

Layer normalization over the trailing dimensions, as torch.nn.functional.layer_norm.

Each row of the trailing normalized_shape dimensions is normalized with its mean and biased variance, (x - mean) / sqrt(var + eps), then scaled by weight and shifted by bias when they are given.

Parameters:

  • x (ndarray of float32) – Input; float64 raises TypeError.

  • normalized_shape (int or tuple of int or None, default None) – Trailing shape to normalize over. None uses the last dimension of x.

  • weight (ndarray or None, default None) – Elementwise scale with normalized_shape values.

  • bias (ndarray or None, default None) – Elementwise shift with normalized_shape values.

  • eps (float, default 1e-5) – Added to the variance inside the square root.

  • numeric_mode ({‘identical’, ‘deterministic’, ‘fast’} or None, default None) – Numeric mode. None uses the library default.

Returns:

y (ndarray of float32) – Normalized output, the shape of x.

Raises:

ValueError – When the trailing dimensions of x do not match normalized_shape, or weight/bias have the wrong size.

Examples

>>> import numpy as np, mojolearn
>>> x = np.random.default_rng(0).random((32, 64), dtype=np.float32)
>>> y = mojolearn.layer_norm_forward(x)
On this page