mojolearn.tokenizer

Byte-level BPE tokenization with a vocabulary you supply or train.

mojolearn ships no vocabulary. BpeTokenizer encodes and decodes byte-level BPE by merge rank, splitting text into pre-tokens with the GPT-2 pre-tokenization pattern, with <|endoftext|> as the id after the last rank. Load a vocabulary you have, or train one with BpeVocabularyTrainer:

tok = BpeTokenizer.from_files("encoder.json", "vocab.bpe")
tok = BpeTokenizer.from_ranks_file("ranks.tsv")      # rank<TAB>hex lines
tok = BpeTokenizer.from_token_bytes(list_of_bytes)   # rank = list index
tok = BpeVocabularyTrainer(vocab_size=32000).train(docs).tokenizer()

With the encoder.json and vocab.bpe files OpenAI publishes with GPT-2 (download them yourself), the ids are GPT-2’s. Tokenization runs in compiled code on the CPU and reads no data file other than the vocabulary. For the Llama 3 and Qwen 2 pre-tokenization patterns, see mojolearn.models.Tokenizer.

GPT2Tokenizer is a deprecated alias of BpeTokenizer (accessing it issues a DeprecationWarning).

Classes

BpeTokenizer([ranks_file])	Byte-level BPE tokenizer in the GPT-2 format, over your vocabulary.
BpeVocabularyTrainer([vocab_size, ...])	Train a byte-level BPE vocabulary deterministically.
TrainedBpeVocabulary(tokens, merges, stats)	A vocabulary trained by BpeVocabularyTrainer.train().

class mojolearn.tokenizer.BpeTokenizer(ranks_file=None)

Bases: object

Byte-level BPE tokenizer in the GPT-2 format, over your vocabulary.

Ids are in [0, n_vocab): the vocabulary’s ranks, then <|endoftext|> at n_vocab - 1. The special token is recognized only with allow_endoftext=True; otherwise its characters are ordinary text. There are no other special tokens.

Bytes are the interface: encode_bytes() and decode_bytes() do the work. encode() also accepts str (encoded as UTF-8 first); decode() decodes the bytes as UTF-8 with errors="replace", since a token sequence may end inside a character. Invalid UTF-8 input round trips.

GPT2Tokenizer is a deprecated alias of this class.

Parameters:

ranks_file (str or path-like) – A rank file (see from_ranks_file()). Required: there is no built-in vocabulary, so omitting it raises ValueError.

Raises:

FileNotFoundError – If ranks_file does not exist.

Examples

>>> from mojolearn.tokenizer import BpeTokenizer
>>> tok = BpeTokenizer.from_files("encoder.json", "vocab.bpe")
>>> ids = tok.encode("Hello, world")
>>> tok.decode(ids)
'Hello, world'

classmethod from_ranks_file(path)

Load a rank file.

One rank<TAB>hex_of_token_bytes line per rank, ascending from 0, with all 256 single bytes present (the format TrainedBpeVocabulary.write_ranks() writes).

Returns:

BpeTokenizer

classmethod from_token_bytes(tokens)

Build a tokenizer from a list of token byte strings.

A token’s rank (and id) is its index. Tokens must be unique and non-empty, and all 256 single bytes must be present; otherwise ValueError is raised before anything is loaded.

Parameters:

tokens (sequence of bytes-like)

Returns:

BpeTokenizer

classmethod from_files(encoder_json, vocab_bpe)

Load the two files of the GPT-2 format.

Parameters:

• encoder_json (str or path-like) – encoder.json: token spelling to id.

• vocab_bpe (str or path-like) – vocab.bpe: the merge list.

Returns:

BpeTokenizer

Raises:

ValueError – If the files are inconsistent: a spelling that does not decode through the byte-to-unicode table, ids that are not 0 to n-1 (with <|endoftext|>, if present, at n), a merge whose parts or result are not tokens, merge results whose ids do not increase above both parts, or a multi-byte token no merge makes. These are the conditions under which merging by rank reproduces the merge list.

property n_vocab

the ranks plus <|endoftext|>.

Type:

Number of ids

property eot_token

The id of <|endoftext|>, n_vocab - 1.

property vocabulary_source

Where the vocabulary was loaded from.

property identity

The vocabulary’s identity, independent of the file it came from.

dict(schema, sha256, n_ranks, n_vocab, endoftext_id), with sha256 over the canonical rank file (rank<TAB>lowercase hex per line). The same table loaded from a rank file, from encoder.json + vocab.bpe or from a trained vocabulary has the same identity. Models trained through mojolearn.lm_corpus record it, so their ids are never decoded with another table.

encode_bytes(data, allow_endoftext=False)

Encode a byte string.

Parameters:

• data (bytes-like) – The input; str raises TypeError (use encode()).

• allow_endoftext (bool, default False) – Recognize <|endoftext|> as the special token.

Returns:

list of int

encode(text, allow_endoftext=False)

Encode text.

Parameters:

• text (str or bytes-like) – str is encoded as UTF-8 first.

• allow_endoftext (bool, default False) – Recognize <|endoftext|> as the special token.

Returns:

list of int

encode_batch(documents, allow_endoftext=False)

Encode several documents in one call.

Each document is encoded alone: encode_batch(docs)[k] == encode(docs[k]), whatever else is in the batch. One call is faster than many for short documents.

Parameters:

• documents (sequence of str or bytes-like)

• allow_endoftext (bool, default False) – Recognize <|endoftext|> as the special token.

Returns:

list of list of int

decode_bytes(ids)

Decode ids to bytes.

Parameters:

ids (sequence of int) – Each in [0, n_vocab); an invalid id raises an error naming its value and position before anything is decoded.

Returns:

bytes

decode(ids, errors='replace')

Decode ids to text.

Parameters:

• ids (sequence of int)

• errors (str, default “replace”) – Passed to bytes.decode.

Returns:

str

decode_bytes_batch(batch)

Decode several id sequences to bytes in one call.

Equivalent to [decode_bytes(ids) for ids in batch]; an invalid id is reported by its position within its own sequence.

Returns:

list of bytes

decode_batch(batch, errors='replace')

Decode several id sequences to text: [decode(ids, errors) for ids in batch].

class mojolearn.tokenizer.BpeVocabularyTrainer(vocab_size=32000, min_frequency=2, backend='auto')

Bases: object

Train a byte-level BPE vocabulary deterministically.

The same corpus and settings give the same vocabulary bytes on any machine: ties are broken by a total order (highest count, then smallest (left_id, right_id)), counting is exact and single-threaded, and no floating point is involved. Training runs on the CPU in compiled code.

Each document is pre-tokenized alone, so no pre-token spans two documents and the order of the documents does not affect the result.

Parameters:

• vocab_size (int, default 32000) – Number of ranks to train, including the 256 single bytes; at least 256.

• min_frequency (int, default 2) – Minimum pair count for a merge; at least 1.

• backend ({“auto”, “mojo”}, default “auto”) – Both name the compiled trainer.

Examples

>>> from mojolearn.tokenizer import BpeVocabularyTrainer
>>> docs = ["the cat sat on the mat", "the dog sat on the log"]
>>> vocab = BpeVocabularyTrainer(vocab_size=300).train(docs)
>>> tok = vocab.tokenizer()
>>> tok.decode(tok.encode("the cat"))
'the cat'

train(documents)

Train on documents.

Parameters:

documents (sequence of str or bytes-like) – At least one document; str is encoded as UTF-8.

Returns:

TrainedBpeVocabulary

class mojolearn.tokenizer.TrainedBpeVocabulary(tokens, merges, stats)

Bases: object

A vocabulary trained by BpeVocabularyTrainer.train().

Variables:

• tokens (list of bytes) – Token byte strings in rank order (rank = id), the 256 single bytes first.

• merges (list of tuple of int) – (left_id, right_id, new_id) in the order the merges were made.

• stats (dict) – Training statistics, including n_ties_broken and tie_break.

property n_tokens

Number of ranks; <|endoftext|> takes the id after the last one.

property n_ties_broken

How many merge selections had two or more pairs at the top count.

property tie_break

The rule that settles equal counts, as text.

render_ranks()

The vocabulary as rank-file text: rank<TAB>hex_of_token_bytes per line.

render_tokenizer_json()

The vocabulary as Hugging Face tokenizer.json text, loadable by the tokenizers library.

write_ranks(path)

Write render_ranks() to path and return path.

write_tokenizer_json(path)

Write render_tokenizer_json() to path and return path.

property identity

The BpeTokenizer.identity of this vocabulary, without loading it.

tokenizer()

A BpeTokenizer over this vocabulary, without writing a file.