mojolearn.GPT2Tokenizer#

mojolearn.GPT2Tokenizer#

Methods

`decode`(ids[, errors])	Decode ids to text.
`decode_batch`(batch[, errors])	Decode several id sequences to text: `[decode(ids, errors) for ids in batch]`.
`decode_bytes`(ids)	Decode ids to bytes.
`decode_bytes_batch`(batch)	Decode several id sequences to bytes in one call.
`encode`(text[, allow_endoftext])	Encode text.
`encode_batch`(documents[, allow_endoftext])	Encode several documents in one call.
`encode_bytes`(data[, allow_endoftext])	Encode a byte string.
`from_files`(encoder_json, vocab_bpe)	Load the two files of the GPT-2 format.
`from_ranks_file`(path)	Load a rank file.
`from_token_bytes`(tokens)	Build a tokenizer from a list of token byte strings.

Attributes

`ENDOFTEXT`
`eot_token`	The id of `<\|endoftext\|>`, `n_vocab - 1`.
`identity`	The vocabulary's identity, independent of the file it came from.
`n_vocab`	the ranks plus `<\|endoftext\|>`.
`vocabulary_source`	Where the vocabulary was loaded from.