Tokens, Context Windows, and Model Limits

Every LLM operates within hard constraints that shape what it can and cannot do. Tokens and context windows are the two most important limits evaluators encounter daily. Understanding them explains many failures that initially look like model incompetence.

What Is a Token?

Models do not read words the way humans do. They process tokens — subword units created by a tokenizer. Common words map to single tokens ("the" = one token). Unusual words split into multiple pieces ("unhappiness" might become "un" + "happiness"). Code, numbers, and non-English text may tokenize unpredictably.

Why it matters: Tokenization affects cost (models are priced per token), speed, and comprehension. A prompt heavy with rare tokens leaves less room for the response within the context window.

Context Windows

The context window is the maximum number of tokens a model can process in a single interaction — prompt plus response combined. Early models handled 2,000-4,000 tokens (roughly 1,500-3,000 words). Modern models support 128,000 to over 1 million tokens.

When input exceeds the context window, the model cannot see the overflow. Information is silently dropped — usually from the middle or beginning, depending on the architecture. This causes failures that look like the model "forgot" earlier instructions or document content.

Common Failure Patterns

Lost instructions. In long conversations, the model may ignore system prompts or early guidelines because they have been pushed out of the active context.

Incomplete document analysis. When asked to summarize a document that exceeds the context window, the model summarizes only what it can see — without indicating that content was truncated.

Inconsistent long-context performance. Some models accept large contexts but attend poorly to information in the middle — the "lost in the middle" phenomenon.

Evaluating Within Limits

When reviewing outputs, check whether the task provided more input than the model could fully process. If a model fails on a 50-page document, determine whether the failure is comprehension or context truncation. This distinction is critical for actionable feedback.

Practical Tips

• Note the approximate length of inputs in your evaluations
• If a model misses information from early in a long prompt, suspect context limits
• Compare whether failures occur on short vs. long inputs
• Report truncation issues separately from reasoning or knowledge failures

Key Takeaways

• Tokens are subword units; tokenization affects cost, speed, and comprehension
• Context windows cap total input plus output length per interaction
• Exceeding context limits causes silent information loss, not explicit errors
• Distinguish context-limit failures from genuine model weaknesses in your reports