1EyeAI

Token

The unit of text a model reads and writes, often a word or a piece of a word.

Text is split into tokens by a tokenizer before a model processes it. Model limits and API prices are usually measured in tokens.

Related: Context window, Large language model (LLM)

One email. Only the signal.

The day's AI news that matters, sourced and dated. Unsubscribe any time.