A Million Tokens, A Thousand Disappointments

The benchmark the model wins, the bill the user pays

Claude Opus 4.6 and Gemini 2.5 Pro both ship with 1M-token context windows; GPT-5.4 also ships 1M (leaked specs for unreleased models hint at 2M but nothing officially advertises that yet). The benchmarks are real — Needle in a Haystack at 99.9% retrieval, MRCR at near-perfect, RULER scores that nobody published in 2024 because nobody could clear 50%. The marketing is all true. And almost nobody runs production workloads at that size.

Pull a survey of teams shipping AI features in 2026 and the median context length they actually use is around 32K-64K tokens. Not because the longer windows don't work — they do. Because the longer windows make three things worse: latency, cost, and the model's effective attention.

The latency cliff

A model's time-to-first-token grows roughly linearly with input length, plus a fixed prefill cost. For Claude Opus at 50K tokens, time-to-first-token is around 1.5 seconds. At 500K, it's 12-15 seconds. At 1M, often 25-35 seconds even on a warm cache.

For a chatbot, that's the difference between "fast" and "broken." For a background agent it might be acceptable, but for any interactive product the latency cliff is the first thing that pushes you back down to 64K.

The cost wall

Long input is the dominant cost in modern LLM applications. At Claude Opus pricing, a single 1M-token request costs around $5 in input alone, before the model has emitted a single output token. A team with 100K daily active users running one such request per session is spending $500K per day. That math kills features before product even sees the demo.

Prompt caching helps — Anthropic's cache hit pricing is roughly 10% of fresh input — but cache hits require the prefix to be stable. The moment your conversation diverges, you're back to paying full freight.

The attention problem (the one nobody talks about)

Even setting aside latency and cost, long contexts have a subtler failure mode: the model gets distracted. The literature now has a name for it: context dilution. Stuff a 1M-token window with 200 documents, only one of which is actually relevant, and the model's accuracy on a question about that document drops measurably — even though it can recite the document verbatim if asked.

The Needle in a Haystack test isn't wrong — it's just not the test that matters. The test that matters is: can the model find the relevant context and ignore the irrelevant context and reason correctly across both? That triple is where the wheels come off.

What teams that ship are actually doing

The pattern that's emerged across teams running real AI in 2026 is roughly this:

Treat the context window as a budget, not a goal. The fact that you can stuff 800K tokens in does not mean you should. The cheapest tokens are the ones you didn't send.

Use long context for one specific class of problem. Long-context excels at: code review across an entire repo, medical-record summarization, legal-document comparison, video transcript analysis. It's bad for: anything where the relevant context is small but the surrounding noise is large.

Pair long context with active retrieval. The 2025 hot take was "RAG is dead because of long context." The 2026 reality is the opposite: RAG and long context are complements. Retrieve the relevant subset, then give the model enough surrounding context to ground its answer. The combination beats either alone.

Cache aggressively. If you have a long static context (a codebase, a manual, a corpus of policies), pin it to the prompt prefix and let prompt caching handle the cost. The dynamic part — the user's question, recent changes — goes at the end. This pattern routinely cuts costs by 70%.

The honest version of the marketing

"1M-token context window" is real, useful, and worth having. But it should be marketed the way bandwidth is marketed for a fiber connection: as a maximum, not an everyday operating point. You don't stream Netflix at 10 Gbps because you can't. You don't load 1M tokens into Claude on every request, even if you can.

The teams that figured this out the fastest stopped chasing the headline number and started asking the right question: what's the smallest context that still gets us the right answer? That number, almost always, is 32K-64K. Build for that. Treat the rest as headroom.