mojolearn.layer_norm_backward#

mojolearn.layer_norm_backward(dy, x, normalized_shape=None, weight=None, bias=None, eps=1e-05, numeric_mode=None)[source]#

Gradients of layer_norm_forward.

Parameters:

  • dy (ndarray of float32) – Gradient of the loss with respect to the output, the shape of x.

  • x (ndarray of float32) – The forward input.

  • normalized_shape (int or tuple of int or None, default None) – Trailing shape normalized over. None uses the last dimension of x.

  • weight (ndarray or None, default None) – The forward’s elementwise scale.

  • bias (ndarray or None, default None) – The forward’s elementwise shift.

  • eps (float, default 1e-5) – The forward’s epsilon.

  • numeric_mode ({‘identical’, ‘deterministic’, ‘fast’} or None, default None) – Numeric mode. None uses the library default.

Returns:

  • dx (ndarray of float32) – Gradient with respect to x.

  • dweight (ndarray or None) – Gradient with respect to weight; None when there is no weight.

  • dbias (ndarray or None) – Gradient with respect to bias; None when there is no bias.

Raises:

ValueError – When dy does not have the shape of x, or shapes do not match normalized_shape.

Examples

>>> import numpy as np, mojolearn
>>> x = np.random.default_rng(0).random((32, 64), dtype=np.float32)
>>> dx, dw, db = mojolearn.layer_norm_backward(np.ones_like(x), x)
On this page