Why a Longer Context Window Won't Fix Your AI Agent

Every few months the headline number gets bigger. One million tokens of context. Two million. The implicit promise is that if the window is large enough, your agent can finally hold the whole problem in its head and reason over it.

Simon Du — Reasoning Chief Scientist at Apodex, the company building verification-centric deep-research agents, and an ML theorist who proved one of the foundational convergence results behind why deep learning works — thinks that promise is pointed at the wrong target. On a recent conversation, he laid out why the context window is rarely the thing standing between you and a correct answer.

His core claim is blunt: a genuinely hard research problem can require more than 10 million tokens to solve. No production model holds that today, and chasing a window big enough to swallow it is the wrong fight.

The bottleneck people think they have

When an agent fails on a complex task, the instinct is to blame memory. It “forgot” the earlier step. It “lost the thread.” So you reach for a model with a longer window and hope the problem fits.

“One of the key bottlenecks is that a single agent or a single larger model cannot hold a very, very long context, say more than one million,” Du explains. The reflex is to grow the window. But the problems he cares about don’t fit in any window you can buy.

“The original problem, it may take a very long time context or a lot of tokens, more than say 10 million tokens to solve,” he says. You cannot scale your way out of that with a single model. The ceiling moves; the problem moves faster.

Divide and conquer is the real unlock

The fix Du describes is older than transformers: decomposition. Break the hard problem into sub-problems. Break those into sub-sub-problems. Keep going until each leaf is small.

“If you basically do divide and conquer, then you can decompose a very hard problem into subtasks,” he says. “Once you decompose a problem, you don’t need a very long context for a single sub agent.”

The example he gives makes it concrete. The original question might be sprawling and open-ended. But the actual input to a single search sub-agent is something like: “find a math book in 1980 about calculus.” That sub-agent doesn’t need a million-token window. It needs a small, precise one — and it does that one job well.

So the arithmetic inverts. The system as a whole still consumes 10M+ tokens across all its agents. But no individual agent ever holds more than a sliver of context. You get the reach of a giant window without needing one to exist.

Why this makes planning the most important skill

If decomposition is the unlock, then the quality of an agent system lives or dies on how well it plans. Du puts decomposition first among the capabilities his team trains for, ahead of search and summarization.

“How do you decompose the problem into subproblems… how to decompose a very hard problem into sub problem, and even you can decompose sub problem into subsub problems — this is a planning or decomposition capability,” he says. His team gathers data on how serious scientists attack a problem — whether to search for more information first or decompose immediately from existing knowledge — and trains the model to behave that way.

That reframes where the engineering effort should go. Most teams pour energy into the reasoning step. Du’s argument is that a lot of the value is upstream, in the cut: a model that decomposes cleanly will beat a model with a longer context window and a worse plan.

The practical takeaway

If your agent is choking on a complex task, resist the urge to shop for a bigger window first. Ask instead whether the problem was ever broken down properly. The leaf tasks your sub-agents actually execute should be small enough that context length stops being the constraint. When they aren’t, the planning layer is where to look — not the model’s spec sheet.

FAQ

Why do AI agents fail on long, complex tasks?

Often because the problem was never decomposed properly, not because the context window was too small. A hard task can require over 10 million tokens end to end — far beyond any single model’s window. Breaking it into small sub-tasks lets each agent work within a manageable context.

Does a longer context window make an AI agent more accurate?

Not on its own. According to Simon Du of Apodex, even a million-plus token window can’t hold the hardest problems, which may need 10M+ tokens. The bigger lever is decomposition — splitting the problem so each sub-agent needs only a small, precise context to do its job well.

What is task decomposition in a multi-agent system?

It’s breaking a hard problem into sub-problems, then sub-sub-problems, until each leaf task is small. A sprawling research question becomes concrete sub-tasks like “find a math book from 1980 about calculus.” Each sub-agent handles one, so the system solves the whole without any single agent holding huge context.

How many tokens does a hard research problem actually need?

Simon Du says a genuinely hard problem can require more than 10 million tokens to solve end to end — across an entire agent team, not one model. No production context window covers that, which is why decomposing the work across sub-agents is more effective than buying a longer window.

Should I prioritize planning or reasoning when building an agent?

Planning often deserves more attention than teams give it. Apodex trains decomposition as a first-class capability, ahead of search and summary, because a clean breakdown determines whether every downstream step succeeds. A model that decomposes well can outperform one with a longer context window and a weaker plan.

How does Apodex handle problems too large for one model’s context?

It decomposes the task across a team of sub-agents, each receiving a small, focused input. The full system may consume 10M+ tokens, but no individual agent holds more than a fraction. This is how a deep-research agent tackles problems that would never fit a single context window.

Is a bigger LLM context window always better?

A larger window helps for some tasks but doesn’t solve hard, open-ended problems that exceed any available context. Past a point, the constraint shifts from window size to how well the problem is broken down. Decomposition, not raw window length, is what scales to the hardest problems.

What’s the difference between context length and decomposition?

Context length is how much text one model can hold at once. Decomposition is how you split a problem so each piece fits a small context. Context length is a hardware-style limit you can’t easily exceed; decomposition is a design choice that lets an agent system solve problems far larger than any single window.

How do you train a model to plan like a scientist?

By gathering data on how serious researchers attack problems — whether to search for more information first or decompose from existing knowledge — and training the model to mirror that judgment. Apodex collects this planning data specifically so its agents decompose hard problems the way a domain scientist would.