Lost in the middle: why a longer context window can hide the answer
Research on how language models use long contexts found a consistent U-shaped accuracy curve — strong at the start and end of a context, and measurably worse in the middle, even for models built for long context.
AI-assisted / research-based
This field note was drafted with AI assistance and synthesizes publicly available research papers and disclosed industry practice on an emerging AI technique. It is not based on confidential deployment data, is not investment, legal, medical, or security advice, and every primary claim links directly to its source so you can verify it yourself.
A longer context window is usually marketed as a straightforward win: more room to include more evidence before the model answers. Research into how models actually use that space tells a more specific — and more actionable — story.
A model that can fit a million tokens is not the same claim as a model that uses all million of them equally well.
The experiment and the curve it produced
Lost in the Middle: How Language Models Use Long Contexts ↗, by Liu and colleagues, tests models on tasks specifically designed to require identifying one piece of relevant information placed at a controlled position within a long input — multi-document question answering and key-value retrieval. The position of the relevant information is the only thing varied across test conditions; the content, the context length, and the task all stay fixed.
The context window fits it. Does the model use it?
The result is a consistent, U-shaped performance curve: accuracy is highest when the relevant information sits near the beginning or the end of the input context, and it degrades — sometimes substantially — when the model has to locate and use information buried in the middle. The paper reports this pattern held even for models explicitly built and marketed for long-context use, which rules out "the model just wasn't trained for long inputs" as the explanation.
Why "it fits" and "it's used well" are different claims
This distinction matters because a context-window specification answers only one question: can this much text be included in a single request without an error. It says nothing about a second, separate question: will the model draw on every part of that included text with equal reliability when producing an answer. The paper's contribution is showing these are empirically different properties that a vendor's stated maximum context length does not, by itself, resolve.
The practical implication is specific rather than vague: a system that "checks a box" by confirming the model accepts a given input length has not verified that the system will actually find and use information anywhere within that length — only that it did not throw an error.
What this changes in production
Retrieval ranking and evidence ordering
Rather than trusting the model to weigh every retrieved passage equally regardless of position, a retrieval pipeline can deliberately place the most relevant or highest-confidence passages near the start or end of the assembled context — the positions the paper's own results show the model uses most reliably.
Put what matters where it gets used
Context-budget discipline over "just include more"
Teams building long-context features often default to including as much retrieved or historical content as the window allows, on the assumption that more context can only help. This research is a direct argument against that default: trimming irrelevant material actively helps, because padding the middle with lower-value content pushes genuinely relevant information into the position where it is least reliably used.
Evaluation design for any long-context feature
Testing a long-context system only with the answer conveniently placed near the edges of the input — which is how many benchmarks and casual demos are structured — will systematically overstate real-world reliability. A rigorous evaluation varies the position of the answer across the full length of the context, the way this paper's own methodology does.
A production checklist
Before shipping a feature that depends on long-context retrieval or reasoning, the team should be able to answer:
- Has accuracy been tested with the relevant information deliberately placed at multiple positions — beginning, middle, and end — not only at the positions a demo happens to use?
- Does the retrieval or context-assembly logic actively order passages by relevance, or does it place them in an arbitrary or purely chronological order?
- Is context budget treated as a scarce resource to curate, or is "include everything that fits" the default policy?
- Has this specific model been tested for this effect, given that the paper's own finding is that the severity varies by model even though the general pattern is consistent?
- For agent or conversation systems with growing context, is older or lower-priority history summarized or trimmed before it drifts into the unreliable middle of the window?
- Does the evaluation suite report accuracy by answer-position, or only an aggregate score that could hide a middle-of-context failure entirely?
Research referenced
- Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts ↗.