mojolearn.cross_entropy#

mojolearn.cross_entropy(logits, targets, ignore_index=-100, reduction='mean', label_smoothing=0.0, num_items=None, return_grad=False, numeric_mode=None, **kwargs)[source]#

Cross-entropy loss over class indices, as torch.nn.functional.cross_entropy.

logits is (N, V) float32 and targets is (N,) int32 or int64 holding a class index or ignore_index. Returns a float for reduction='mean' or 'sum' and an (N,) float32 array for 'none'. With return_grad=True the forward and the gradient with respect to the logits come from one call: (loss, dlogits), dlogits of shape (N, V).

Parameters:

  • logits (buffer, shape (N, V)) – float32 logits, row-major.

  • targets (buffer, shape (N,)) – Integer class indices; a value that is neither a valid class nor ignore_index raises an error naming its row.

  • ignore_index (int, default -100) – Targets equal to this are ignored, even when it is a valid class index (as torch).

  • reduction ({‘mean’, ‘sum’, ‘none’}, default ‘mean’) – Reduction over rows. 'none' has no gradient: return_grad=True with it is refused.

  • label_smoothing (float, default 0.0) – Label smoothing in [0, 1).

  • num_items (int or None, default None) – Divisor of the mean, as Hugging Face’s num_items_in_batch. None uses the number of non-ignored targets.

  • return_grad (bool, default False) – Also return the gradient with respect to logits.

  • numeric_mode ({‘identical’, ‘deterministic’, ‘fast’} or None, default None) – Numeric mode. None uses the library default.

Returns:

  • loss (float or ndarray) – The loss; (N,) per-row values under reduction='none'.

  • dlogits (array, shape (N, V)) – Only with return_grad=True.

Notes

Refused: per-class weight, the deprecated size_average and reduce, class-probability (soft) targets, float16/bfloat16 and float64. Memory use is several (N, V) float32 buffers; for a large vocabulary, split the rows into chunks (with reduction='none' the chunked result equals the unsplit one).

Examples

>>> import numpy as np, mojolearn
>>> logits = np.random.default_rng(0).random((32, 10), dtype=np.float32)
>>> targets = np.arange(32, dtype=np.int64) % 10
>>> loss, dlogits = mojolearn.cross_entropy(logits, targets, return_grad=True)
On this page