A token is a chunk of text, often part of a word, that a language model reads and generates.
Why it matters
As a rough guide, one token is about four characters of English. Context limits and prices are counted in tokens.
A token is a chunk of text, often part of a word, that a language model reads and generates.
As a rough guide, one token is about four characters of English. Context limits and prices are counted in tokens.