BROWSEWIZ: LESSONS FROM BUILDING A PRODUCTION BROWSER AGENT

Introduction

The uncomfortable truth about most "AI products": they are a fetch() call to OpenAI, a few prompt strings, and a React component. That works for demos. It falls apart in production.

BrowseWiz is a Chrome extension, built and shipped by DevPeer, that browses the web and automates tasks in your browser tabs for you — it can also chat about any webpage. It ships with built-in tools for Google Sheets and Google Calendar, and can be extended with custom tools, remote MCP servers, and any OpenAI API-compatible model endpoint (Chat Completions API), all backed by a quota-managed cloud backend. It's live on the Chrome Web Store today. The codebase includes a browser extension, an Express API, Lambda functions, shared libraries, and a Next.js marketing site.

Building it required solving five problems that most AI product teams only discover after they've shipped. This is what we learned.

1. Agentic Loops Are State Machines, Not Prompt Chains

The biggest mistake teams make when building agents is treating them as a one-shot request/response. Send a message, get a reply. Simple. The problem is that a capable agent will call tools, receive results, reason about them, and call more tools — potentially for many iterations before it produces a final answer. If your architecture assumes one turn, it will fail in ways that are genuinely hard to diagnose.

The temptation is to cap the system at a single tool call and call it an "agent." But users immediately find tasks where that isn't enough — "summarise these three links and add the key points to my notes" requires web fetching, content extraction, and a write operation. The multi-step complexity is not optional. It's load-bearing.

BrowseWiz runs a main chat loop — user message → LLM generation → parallel tool execution → repeat — with a nested browser automation sub-agent that runs its own ReAct loop capped at 14 iterations. Three problems emerged quickly once this was in production:

Interruption. If a user sends a new message while the agent is mid-loop, you cannot let the old loop's final response overwrite the new one. Without explicit interruption tracking, users see responses from previous queries arrive out of order. The fix is an interruption check before every state update.

Contract enforcement. If the LLM's tool response doesn't meet the expected output format — and it won't, regularly — the choices are: silently fail, hallucinate state, or retry with a corrective prompt. The correct choice is the third one. You have to design the retry into the loop, not treat it as an exceptional case.

Graceful degradation. At the iteration cap, the agent doesn't just stop. It asks the LLM to summarise the work done so far and returns a coherent answer. Users get a result, not a wall of JSON or a silent timeout.

The lesson: Designing an agent means designing a state machine with explicit failure modes at each transition. Hoping the LLM will terminate cleanly is not a design.

2. Context Window Management Is a Product Decision

There's a tempting approach to context management: throw more tokens at the problem. Use the 128K context model, then the 200K model, pay whatever it costs. This works until your most engaged user hits a two-hour research session and you're spending $4 per conversation turn — or until the model's quality noticeably degrades on inputs that long.

The real trade-off is between cost, quality, and information retention. Aggressive compaction keeps cost low and model attention high, but loses conversation detail. Too-conservative compaction is expensive and unpredictable. The compaction threshold is fundamentally a product decision — it just looks like an engineering parameter.

BrowseWiz triggers automatic compaction at 45% of the model's context limit. That threshold exists for a specific reason: it provides enough remaining context to complete the current task before the next compaction window, avoiding the awkward case where the agent runs out of space mid-task. When compaction fires, it calls the LLM to generate a conversation summary, stores the summary alongside an offset pointer, and continues seamlessly from the summary in subsequent turns.

Two constraints shaped the implementation in ways that weren't obvious up front:

The OpenAI API rejects a conversation where a tool call message is not immediately followed by its result. This means you cannot compact at an arbitrary message boundary — the compaction logic has to detect when the last message in the history is a pending tool call and hold it back until the result arrives.

Browser automation sessions have a separate problem: individual tool results (DOM extractions, full-page content) can be enormous and they arrive within a single turn. Compaction handles inter-turn bloat; a separate auto-redaction step prunes oversized tool results mid-session to reclaim token budget in real time.

The lesson: Context window management sits at the intersection of unit economics, model quality, and user experience. Decide on the trade-offs deliberately, not by accident.

3. Browser Automation Breaks on the Assumption That the Web Is Predictable

Browser automation is one of those areas where the first 80% takes a weekend and the last 20% takes months. Finding an element and clicking it works on static pages in controlled environments. It fails comprehensively on modern single-page apps, shadow DOMs, dynamically generated IDs, and elements that are visually present but technically non-interactive.

The tempting shortcut is CSS selectors with a documented caveat that the tool "occasionally fails." The cost of that decision is immediately visible to users: they watch the AI click the wrong button and then confidently report the task is complete.

BrowseWiz uses a hash-based element registry instead. Every interactive element is assigned a 7-character UUID at discovery time. The AI agent references elements by hash, not by selector. This eliminates an entire class of selector-fragility failures — when the DOM changes, the agent re-discovers elements and gets fresh hashes; there are no stale selectors to repair.

Determining whether an element is even clickable required a 10-tier detection algorithm, because no single signal is reliable across the open web. The algorithm works through naturally interactive tags, ARIA roles, event handler attributes, CSS class heuristics, and computed styles — including recursive traversal into shadow DOM trees — before making a determination.

The more important architectural decision was the feedback layer. After every interaction, the agent receives a structured result that answers four questions: Did the DOM change meaningfully? Did the page navigate? Was a form submission detected? If nothing happened, what should the agent try instead? This feedback loop is what allows the agent to detect when it's stuck — clicking an element that produces no effect — and switch strategy, rather than repeating the same action with increasing confidence.

There's also a trust dimension. Browser automation acts on real user accounts with real consequences. BrowseWiz requires explicit approval before acting on any untrusted domain — per-action, per-domain for the session, or globally. Users can whitelist trusted sites. This turns out to be a feature users trust, not a friction point they work around.

The lesson: The engineering effort in browser automation is almost entirely in the observation and feedback layer. The action itself is trivial.

4. Multi-Provider Reliability Is a Routing Problem, Not a Monitoring Problem

Every AI product that relies on a single LLM provider is one capacity event away from an incident. The standard response is to add monitoring and alerts. The better response is to design the system so that a provider going down is handled automatically, without a human in the loop.

The objection to multi-provider routing is complexity: you have to normalize across different APIs, pricing models, token counting conventions, and error formats. That's real. But single-provider dependency is a reliability and negotiating-position risk that compounds as your user base grows. The complexity is worth taking on early.

BrowseWiz uses a proxy model pattern. Virtual model IDs — browsewiz-browse, browsewiz-chat — map to rotating lists of real provider-model pairs. The backend maintains a round-robin index per proxy ID. On a rate limit (HTTP 429), it advances the index and retries with the next provider-model in the list.

There's a concurrency problem that isn't obvious until you hit it: if two requests arrive simultaneously and both get rate-limited, naive shared-state retry logic causes them to both advance the index and retry the same next model — which is already rate-limited. BrowseWiz takes a snapshot of the model list at request start time, so each in-flight request manages its own retry sequence independently. No mutex, no coordination, no race condition.

All three providers — OpenAI, Google Gemini, OpenRouter — are behind a single streaming client implementation using the OpenAI SDK with a custom baseURL. Adding a new provider is a one-line config change.

The lesson: Fault tolerance in LLM-backed systems is an architecture decision. By the time it becomes an ops problem, it's too late to fix cheaply.

5. Streaming and Tool Calling Are Not Designed to Coexist

Streaming tokens to the UI feels like an obvious choice. Users see the response forming in real time rather than staring at a spinner. Tool calling feels equally obvious — agents need to take actions. The problem is that these two features were designed independently, and combining them requires deliberate work at the protocol boundary.

The clean alternative is to skip streaming: wait for the full completion, then display it. Tool call handling becomes trivial. But for longer responses, the UI feels broken — users have learned to expect progressive rendering from every other text interface they use. Regressing on this is a UX cost that's hard to justify once you've seen the difference.

Streaming tokens works via incremental server-sent events — each delta arrives as a small chunk and updates the UI immediately. Tool calls, however, do not arrive as incremental deltas. They arrive as a complete JSON object at the end of the stream. This means the stream parser has to maintain two separate channels: accumulate text deltas for real-time rendering, then reconstruct the full tool call object from the final completion payload. These are handled differently and at different times in the streaming lifecycle.

On the backend, SSE response headers are initialized lazily — only when the first token actually arrives. This means non-streaming requests don't pay the overhead of SSE setup, and the same handler serves both streaming and non-streaming callers without branching.

The lesson: When two protocol-level features interact, the integration work belongs in your architecture. It will not resolve itself at runtime.

What This Means for Your AI Project

None of these five problems are unique to BrowseWiz. They appear in every agentic AI system that reaches production scale, and they appear in roughly this order: the agent loop breaks first, then context overflows, then automation reliability degrades, then provider outages cause incidents, then streaming and tool calling fight each other.

The engineering required to solve them is learnable. What's harder to replicate is having already made the expensive mistakes — the production incident from missing interruption handling, the cost spike from unmanaged context growth, the user complaint about the AI clicking the wrong element.

Agentic AI products fail not because the underlying models are bad. They fail because the surrounding system — the state machine, the context budget, the observation feedback loop, the provider routing, the streaming protocol — wasn't designed for the edge cases that real users find on day one.

The right hiring signal for an agentic AI project is not "has used the OpenAI API." It's "has shipped a system where those five problems were production bugs, not hypothetical concerns."

If you're building a product in this space, I'd be glad to talk.