mojolearn.BpeTokenizer

class mojolearn.BpeTokenizer(ranks_file=None)

Bases: object

Byte-level BPE tokenizer in the GPT-2 format, over your vocabulary.

Ids are in [0, n_vocab): the vocabulary’s ranks, then <|endoftext|> at n_vocab - 1. The special token is recognized only with allow_endoftext=True; otherwise its characters are ordinary text. There are no other special tokens.

Bytes are the interface: encode_bytes() and decode_bytes() do the work. encode() also accepts str (encoded as UTF-8 first); decode() decodes the bytes as UTF-8 with errors="replace", since a token sequence may end inside a character. Invalid UTF-8 input round trips.

GPT2Tokenizer is a deprecated alias of this class.

Parameters:

ranks_file (str or path-like) – A rank file (see from_ranks_file()). Required: there is no built-in vocabulary, so omitting it raises ValueError.

Raises:

FileNotFoundError – If ranks_file does not exist.

Examples

>>> from mojolearn.tokenizer import BpeTokenizer
>>> tok = BpeTokenizer.from_files("encoder.json", "vocab.bpe")
>>> ids = tok.encode("Hello, world")
>>> tok.decode(ids)
'Hello, world'

__init__(ranks_file=None)

Methods

decode(ids[, errors])	Decode ids to text.
decode_batch(batch[, errors])	Decode several id sequences to text: [decode(ids, errors) for ids in batch].
decode_bytes(ids)	Decode ids to bytes.
decode_bytes_batch(batch)	Decode several id sequences to bytes in one call.
encode(text[, allow_endoftext])	Encode text.
encode_batch(documents[, allow_endoftext])	Encode several documents in one call.
encode_bytes(data[, allow_endoftext])	Encode a byte string.
from_files(encoder_json, vocab_bpe)	Load the two files of the GPT-2 format.
from_ranks_file(path)	Load a rank file.
from_token_bytes(tokens)	Build a tokenizer from a list of token byte strings.

Attributes

ENDOFTEXT
eot_token	The id of <|endoftext|>, n_vocab - 1.
identity	The vocabulary's identity, independent of the file it came from.
n_vocab	the ranks plus <|endoftext|>.
vocabulary_source	Where the vocabulary was loaded from.

classmethod from_ranks_file(path)

Load a rank file.

One rank<TAB>hex_of_token_bytes line per rank, ascending from 0, with all 256 single bytes present (the format TrainedBpeVocabulary.write_ranks() writes).

Returns:

BpeTokenizer

classmethod from_token_bytes(tokens)

Build a tokenizer from a list of token byte strings.

A token’s rank (and id) is its index. Tokens must be unique and non-empty, and all 256 single bytes must be present; otherwise ValueError is raised before anything is loaded.

Parameters:

tokens (sequence of bytes-like)

Returns:

BpeTokenizer

classmethod from_files(encoder_json, vocab_bpe)

Load the two files of the GPT-2 format.

Parameters:

• encoder_json (str or path-like) – encoder.json: token spelling to id.

• vocab_bpe (str or path-like) – vocab.bpe: the merge list.

Returns:

BpeTokenizer

Raises:

ValueError – If the files are inconsistent: a spelling that does not decode through the byte-to-unicode table, ids that are not 0 to n-1 (with <|endoftext|>, if present, at n), a merge whose parts or result are not tokens, merge results whose ids do not increase above both parts, or a multi-byte token no merge makes. These are the conditions under which merging by rank reproduces the merge list.

property n_vocab

the ranks plus <|endoftext|>.

Type:

Number of ids

property eot_token

The id of <|endoftext|>, n_vocab - 1.

property vocabulary_source

Where the vocabulary was loaded from.

property identity

The vocabulary’s identity, independent of the file it came from.

dict(schema, sha256, n_ranks, n_vocab, endoftext_id), with sha256 over the canonical rank file (rank<TAB>lowercase hex per line). The same table loaded from a rank file, from encoder.json + vocab.bpe or from a trained vocabulary has the same identity. Models trained through mojolearn.lm_corpus record it, so their ids are never decoded with another table.

encode_bytes(data, allow_endoftext=False)

Encode a byte string.

Parameters:

• data (bytes-like) – The input; str raises TypeError (use encode()).

• allow_endoftext (bool, default False) – Recognize <|endoftext|> as the special token.

Returns:

list of int

encode(text, allow_endoftext=False)

Encode text.

Parameters:

• text (str or bytes-like) – str is encoded as UTF-8 first.

• allow_endoftext (bool, default False) – Recognize <|endoftext|> as the special token.

Returns:

list of int

encode_batch(documents, allow_endoftext=False)

Encode several documents in one call.

Each document is encoded alone: encode_batch(docs)[k] == encode(docs[k]), whatever else is in the batch. One call is faster than many for short documents.

Parameters:

• documents (sequence of str or bytes-like)

• allow_endoftext (bool, default False) – Recognize <|endoftext|> as the special token.

Returns:

list of list of int

decode_bytes(ids)

Decode ids to bytes.

Parameters:

ids (sequence of int) – Each in [0, n_vocab); an invalid id raises an error naming its value and position before anything is decoded.

Returns:

bytes

decode(ids, errors='replace')

Decode ids to text.

Parameters:

• ids (sequence of int)

• errors (str, default “replace”) – Passed to bytes.decode.

Returns:

str

decode_bytes_batch(batch)

Decode several id sequences to bytes in one call.

Equivalent to [decode_bytes(ids) for ids in batch]; an invalid id is reported by its position within its own sequence.

Returns:

list of bytes

decode_batch(batch, errors='replace')

Decode several id sequences to text: [decode(ids, errors) for ids in batch].