mojolearn.OneCycleLR#

class mojolearn.OneCycleLR(max_lr, total_steps, pct_start=0.3, anneal_strategy='cos', div_factor=25.0, final_div_factor=10000.0, three_phase=False)[source]#

Bases: _LrTable, _Sched

One-cycle learning-rate schedule, as torch’s OneCycleLR.

The rate rises from max_lr / div_factor to max_lr over the first pct_start of total_steps, then anneals down to initial_lr / final_div_factor (with three_phase=True: back down to the initial rate, then to the final one), by cosine or linear annealing. lr_at(t) takes the one-based optimizer step and is defined up to total_steps + 1; later steps raise ValueError, as torch does. Each value is the exact closed form rounded once to float32 (ties to even, values below the smallest normal flushed to +0.0), so a schedule gives the same bits on every machine.

Pass it as lr_schedule= to RMSprop, Adagrad, Lion, Adamax, NAdam and the recurrent estimators.

Parameters:

  • max_lr (float) – Peak learning rate. Must be finite.

  • total_steps (int) – Number of steps in the cycle. Must be >= 1.

  • pct_start (float, default 0.3) – Fraction of the cycle spent increasing the rate, in [0, 1].

  • anneal_strategy ({‘cos’, ‘linear’}, default ‘cos’) – Annealing shape.

  • div_factor (float, default 25.0) – Initial rate is max_lr / div_factor.

  • final_div_factor (float, default 1e4) – Final rate is initial_lr / final_div_factor.

  • three_phase (bool, default False) – Use torch’s three-phase form.

Notes

Momentum cycling (cycle_momentum, base_momentum, max_momentum), per-group learning rates and last_epoch resumption are not supported.

Examples

>>> import mojolearn
>>> sched = mojolearn.OneCycleLR(max_lr=0.01, total_steps=1000)
>>> lrs = sched.lrs(1000)
__init__(max_lr, total_steps, pct_start=0.3, anneal_strategy='cos', div_factor=25.0, final_div_factor=10000.0, three_phase=False)[source]#

Methods

bits_at(t)

lr_at(t)

The float32 learning rate of ONE-BASED step t, as a float.

lr_values(n)

lr_at(1) .

lrs(n)

lr_at(1) .