mojolearn.Adagrad#

class mojolearn.Adagrad(params, lr=0.01, lr_decay=0.0, weight_decay=0.0, initial_accumulator_value=0.0, eps=1e-10, numeric_mode=None)[source]#

Bases: _SeqOptimizer

Adagrad optimizer, as torch.optim.Adagrad.

Accumulates squared gradients, sum += g**2, and updates p -= clr * g / (sqrt(sum) + eps) with clr = lr / (1 + (t - 1) * lr_decay). Defaults are torch’s.

Parameters:

  • params (ndarray or list of ndarray) – Parameters to update in place.

  • lr (float, default 1e-2) – Learning rate.

  • lr_decay (float, default 0.0) – Learning-rate decay per step.

  • weight_decay (float, default 0.0) – L2 penalty added to the gradient.

  • initial_accumulator_value (float, default 0.0) – Starting value of the squared-gradient sum.

  • eps (float, default 1e-10) – Term added to the denominator.

  • numeric_mode ({‘identical’, ‘deterministic’, ‘fast’} or None, default None) – Numeric mode of the update. None uses the library default.

Variables:

  • t (int) – Number of steps taken.

  • sum (ndarray) – Accumulated squared gradients, flat over all parameters.

Notes

params must be C-contiguous float32 NumPy arrays, updated in place; anything else raises TypeError. step(grads) takes one gradient per parameter, in the same order and shapes; float64 gradients raise TypeError. lr is read at every step, so you can change opt.lr between steps, or set opt.lr_schedule to a schedule (StepLR, ExponentialLR, OneCycleLR, or any object with a one-based lr_at(t)), which sets lr before each step. state_dict() and load_state_dict() save and restore the step count, learning rate and state buffers.

Not supported: parameter groups, maximize, foreach/fused/capturable/ differentiable, sparse gradients and float64 parameters.

Examples

>>> import numpy as np, mojolearn
>>> w = np.zeros(128, dtype=np.float32)
>>> opt = mojolearn.Adagrad(w, lr=0.1).step(np.ones_like(w))
__init__(params, lr=0.01, lr_decay=0.0, weight_decay=0.0, initial_accumulator_value=0.0, eps=1e-10, numeric_mode=None)[source]#

Methods

load_state_dict(sd)

state_dict()

step(grads)

One update of every param from grads (same shapes, same order).

Attributes

resident_

True once the state lives on the device.

state

sum

Accumulated squared gradients, flat over all parameters.
property sum#

Accumulated squared gradients, flat over all parameters.