Why Edge Cases Kill Browser Automation Projects — And How to Fix Them

When you’re building a browser automation system from scratch, the first version seems simple: take a screenshot, feed it to an LLM, ask where to click, send coordinates back to the browser, repeat. In theory, that works. In practice, that approach breaks in production about 50% of the time.

Magnus Müller, CEO of Browser Use, has spent the last year scaling a platform that does this reliably. When he talks about why naive implementations fail, he’s not theorizing — he’s seen thousands of real user tasks fail in ways that a simple loop can’t handle.

The Problem Is Invisible Until It’s in Production

The screenshot-to-LLM approach works beautifully for happy paths. You want to buy something on Amazon? Click the search box, type the product, click search, click the product, add to cart, checkout. Straightforward.

But here’s what actually happens in the real world: “Suddenly you need like a confirmation, a 2FA confirmation. Oh, you need to log in. Oh, you need to have a persistent login. Oh, LinkedIn blocks you and they think you’re a bot. No, Amazon thinks you’re a bot and they block you.”

Each of these moments requires a different handling strategy. Two-factor authentication isn’t just a click — it’s a handoff to a different system. Bot detection requires the agent to understand it’s blocked and handle it gracefully. Cross-origin iframes (like when Stripe payment processing happens inside Amazon’s UI) look like normal interaction to an end user but require the agent to understand it’s actually interacting with a different domain entirely.

Magnus puts it plainly: “If 50% of the time that my workflow doesn’t work and it would just give me an error and then I don’t feel it’s very useful, right. How much do you think actually edge cases takes up the whole, you know, the, the incidents?”

The answer: nearly all of them. “I think it’s full of edge cases.”

What Makes Production Browser Agents Different

The gap between a working demo and a working system isn’t small tweaks — it’s comprehensive edge case handling. “Every single task we see has some form of an edge case,” Magnus explains. The variations pile up:

• Authentication flows: Persistent cookies, session management, 2FA via email or SMS
• Bot detection: Captchas, rate limiting, user-agent blocking
• DOM complexity: Iframes, shadow DOM, dynamically loaded content
• File operations: Upload handling without exposing credentials to the LLM
• Scroll context: Some elements only appear when you scroll inside a specific container, not the main page
• Cross-site payments: When third-party payment processors embed themselves in checkout flows
• Site variability: When what works on desktop breaks on mobile, or the site changes between runs

This isn’t a long tail — it’s the distribution. Magnus saw this clearly when he asked himself: “If I want to build a browser automation agent from scratch… what more can I get other than what I did?” The answer is everything else.

The Real Cost of Edge Cases

In Magnus’s system, the costs are measurable. An evaluation run with 100 tasks costs approximately $60 using Claude. If edge case handling is poor, you’re burning tokens on failed retries and hallucinated workarounds. If it’s robust, the agent knows when to stop, ask for clarification, or escalate.

The infrastructure layer has to make decisions that simple LLM loops can’t make. “Do you want to have it like a little bit faster? Or you want to block certain domains? Or you want to sign suddenly a field?” These are all edge cases that seem minor until they’re blocking your task.

One concrete example Magnus shared: “I think I would love if like every day I would have someone to check this for me. But of course I don’t do it myself because it’s just not the most important thing.” A user came to him with a real case: “Last year in July, their solar roof broke. Okay. And then they lost like for two months, their energy savings because there was like a, in a pipeline, it was a broken thing. And he could immediately, he would have seen it in his dashboard if there’s something broken, but he just never checked his dashboard.”

A simple agent would check the dashboard, screenshot it, and fail if the layout didn’t match expectations or if authentication had expired. A production system has to handle dashboard authentication, understand what “something broken” means on a solar monitoring interface, and alert the user only when there’s an actual problem.

How to Handle Them

The naive approach to edge cases is to hard-code them: “If captcha detected, stop. If 2FA required, ask user.” This works for specific cases but creates fragile, unmaintainable code.

Magnus’s approach is different. Instead of making edge case handling reactive, he’s made it structural. By separating the LLM’s task into “exploration” and “execution” phases, with a meta-learning system that stores what worked, the agent can handle variations without bespoke code.

The system works like this: when an agent encounters an edge case, it adapts. If it finds a workaround (like clicking a “Remember this device” button to bypass 2FA), that workaround gets stored. The next time the agent encounters that site, it already knows to look for that button. “You compress all of them into this one markdown file as a learning. And the next time the agent comes, you can skip this computation, what you compress into this learning file.”

This transforms edge cases from failures into learning opportunities. The agent doesn’t need a human to manually code every variation — it learns from its own exploration.

Why Your Simple Approach Will Fail

If you’re thinking “I’ll just clone the Browser Use repo or use Playwright with a screenshot loop,” you’re outsourcing the edge case problem. You’re not solving it yourself — you’re delegating it. That’s actually the right call, because solving edge cases comprehensively is not a weekend project.

Magnus’s honest assessment: “I think over time, those agents will get simpler and simpler. You will have the base LLM, which is very good at screenshot understanding. You have a simple tool to click on coordinates like a human and type like a human. Like in the end, if you just give a mouse and a keyboard, in theory, it should be able to do everything like a human can.”

But between now and when that’s true, you need systems that understand the complexity of the web as it exists today — with all its authentication patterns, bot detection, payment processors, and layout variations.

FAQ

Why does my screenshot + LLM approach fail so often?

The approach itself is sound, but production web tasks have too many variations for an LLM to handle without infrastructure support. Persistent authentication, bot detection, cross-site iframes, and fallback flows all require systems-level handling, not just prompting tricks. Most tasks fail on these edge cases, not on the core logic.

What’s the most common edge case that breaks browser automation?

Authentication and persistence. Users expect agents to remember login state, handle 2FA gracefully, and navigate around bot detection. A naive screenshot loop treats each step as stateless, which breaks the moment you need a persistent session or an out-of-band confirmation (email verification, SMS code, etc.).

How do I know if an edge case is worth handling?

If it blocks more than one user task or if it appears across multiple websites (like captchas or 2FA), it’s worth building infrastructure for it. If it’s site-specific, consider whether your meta-learning approach can discover the pattern instead of you hard-coding it.

Can LLMs detect edge cases themselves?

Partially. An LLM can recognize “I see a captcha” or “I see a login page” if you’ve given it examples. The challenge is knowing what to do about it. That’s where structured error handling and meta-learning come in — not just recognition, but adaptation.

Do I need a framework like Browser Use, or can I build edge case handling myself?

You can build it, but it’s expensive. Magnus’s team has spent the year systematizing the patterns they’ve seen across thousands of tasks. If you have the time and tokens to spend, you can learn these patterns by running your own agents. Most teams choose to use a framework that’s already learned them.

How much does it cost to run evaluations that catch edge cases?

It depends on the model and task complexity. Magnus runs 100-task evaluation sets that cost $30-60 per run using Claude. He runs multiple cycles a day during active development. The tradeoff: spending on evaluation upfront saves you from shipping broken agents to users.

If my agent hits an edge case, how should it respond?

Ideally, the agent either recovers (by following a fallback pattern it learned before) or fails gracefully with a clear reason. “I see a captcha and cannot proceed” is better than “error” or silent failure. This is why meta-learning systems that document what worked and what didn’t are valuable — they let the agent make intelligent choices, not just guess.