Language models and tokenizers

Language-model training and inference, checkpoints, tokenizers and corpora.

BpeTokenizer	Byte-level BPE tokenizer in the GPT-2 format, over your vocabulary.
ByteLanguageModelConfig	Dimensions of the decoder language model: batch, sequence, layers, vocabulary.
GPT2Tokenizer
language_model	Train and run FP32 decoder language models.
LanguageModelConfig
LanguageModelInference	Run a trained decoder language model on the CPU, with no GPU.
LanguageModelTrainer
lm_corpus	Tokenize a training corpus once and train a language model on the ids.
models	Load and run Hugging Face-format causal language models.
SmallByteLanguageModelTrainer	FP32 decoder language model trainer (AdamW, mean cross-entropy).
tokenizer	Byte-level BPE tokenization with a vocabulary you supply or train.