We build AI-powered performance testing tools. Our platform already had a solid foundation: an AI-driven correlation workflow that analyses test recordings, identifies dynamic values, and wires up the extractors needed to make tests repeatable. That core pipeline worked. It had bugs — what doesn't? — but the workflow was sound, and users relied on it daily.
Alongside that, we had a conversational AI agent. Think of it as an expert advisor that could look at your test results, explain what went wrong, and suggest what to do next. It was useful. Users liked it. It gave good advice.
Then we got ambitious.
What if the agent didn't just advise — what if it could autonomously diagnose failures across an entire test plan, reason about which extractors were broken and why, and directly mutate the test configuration to fix them? Not one fix at a time through a chat interface, but full-spectrum analysis: ingest the world view, form a diagnosis, execute a multi-step repair plan.
God mode.
The remarkable thing is that it worked — when we threw the most powerful (and expensive) model at it. Opus could look at a complete test plan, reason about cascading extraction failures, and produce correct multi-step fixes. The concept was proven. But it was slow, expensive, and only reliable with the heaviest model available. A sledgehammer to crack a nut.
We didn't scale back the ambition. We found a different path to get there.
The Gap Between "It Works on Opus" and "It Works in Production"
Five pathways to affordable god mode
To make god mode work without Opus-level costs, we needed smart routing. Not every query needs the flagship model. "What does this error mean?" is a different problem from "diagnose and fix my entire test plan."
So we built a triage layer with routing pathways: canned responses for simple greetings, clarification requests for ambiguous queries, an efficient executor using a cheaper model without tools, the full flagship executor with all capabilities, and code-driven shortcuts for structural queries like "how many steps?".
On paper, this would give us Opus-quality results where it mattered and fast, cheap responses everywhere else.
In practice, the classification boundaries were where things fell apart. The "efficient" path used a smaller model without tool access. When the triage model misrouted a mutation task, the agent would describe the fix in plain text instead of executing it. Confident-sounding responses that did nothing. The user had no idea the agent couldn't do what it was describing.
Canned clarifications were brittle. Keyword matching needed constant manual maintenance. Each pathway created new edge cases at its boundaries with the others.
We hadn't broken the product — our core workflow kept running, and other agents kept working fine. But the new god-mode capability remained unreliable.
OpenAI-compatible is a spectrum
Trying to reduce costs by running on different LLM providers taught us that API compatibility isn't binary.
The sequential tool caller. One popular open-source model returned exactly one tool call per response. Our agent needed eight tools for a full diagnosis. Eight round-trips at 15–20 seconds each: nearly three minutes of spinner, 99.7% of which was pure LLM wait time. The model wasn't slow — it refused to batch.
The parameter that worked everywhere except here. When we switched to a different provider, our triage layer sent reasoning_effort: "none" — a standard OpenAI parameter, silently ignored by the previous provider. The new one rejected it with a 400. But the error was caught, the main dispatch continued fine, while a callback the HTTP layer was waiting on was never signalled. Result: 500 "Triage timeout" on every request — for a dispatch that was completing successfully in the background. Other agents that bypassed triage worked perfectly. Only the god-mode path was broken.
Two bugs, one parameter, a 500 on a request that was succeeding. The kind of compound failure you only discover in production with a real provider.
Prompt archaeology
As we iterated on the agent's prompt, we discovered it existed in six different places: a file in the repo, a bind-mounted directory on the server, a copy in a separate routing service, inline strings in the dispatch code, a backup in a config store, and an archived version. Update one, see no change in production, wonder if you're losing your mind.
Not a design flaw — the natural accumulation of a fast-moving codebase. But it meant we couldn't reliably improve the agent's behaviour because we couldn't be sure which prompt it was using.
Finding the Smarter Path
We never scaled back what we wanted the agent to do. We changed how it does it.
The original trial was brute-force god mode: throw the entire test plan at the biggest model, let it reason about everything, call lots of tools, and make all the fixes. Opus could do this, but it was slow and expensive.
The breakthrough was realising that the agent doesn't need to do everything itself. It needs to orchestrate.
Delegate, don't do
Instead of the agent independently researching, diagnosing, and fixing in one expensive multi-turn session, it now leans into the specialised agents and workflows we'd already built. Need to check the correlation health? Trigger a correlation run — it takes seconds to kick off and returns structured results. Need to QA the test plan? Trigger that workflow. The agent coordinates existing capabilities rather than reimplementing them through tool calls.
Direct mutations still happen — the agent can still modify test configurations — but as a targeted action after diagnosis, not as step twelve of a sprawling reasoning chain. And because each specialised workflow is optimised for its specific job, the combined result is faster and more reliable than one model trying to do everything.
Move with confidence, roll back instantly
The old approach involved a lot of back-and-forth: the agent would research, ask the user for confirmation, research more, propose a change, wait for approval, and make the change. Safe, but painfully slow.
By investing in high-quality world view data upfront — pre-hydrated context from multiple sources, freshness-verified before dispatch — the agent now has what it needs from the first turn. It can move straight to analysis and action. No wasted turns digging out data.
And because we built accept/reject rollback capability into every mutation, confidence doesn't require caution. The agent says, "I've fixed it," and if it got it wrong, the user rejects the change and rolls back to exactly where they were. No damage done. This changes the entire interaction model: from "may I?" to "here you go, keep it or toss it."
Smart routing (take two)
We replaced the five-pathway triage with two outcomes: direct (the fast model handles it entirely) and execute (full dispatch with all tools and capabilities). That's it.
The critical design property: every execution path has tools. There is no execution path without tool access. The entire class of "misrouted to a toolless executor" bugs became structurally impossible. Not fixed — impossible.
For clear-cut intents — greetings, simple product questions — a keyword fast-path handles them without any LLM call at all. Fifteen lines of code. The simplest code in our system became the most reliable routing layer.
Pre-hydrate, don't round-trip
The triage call now emits a context_needs declaration alongside its routing decision. Before the flagship model starts, the server pre-executes the relevant discovery tools and assembles the context. The expensive model gets everything it needs on its first turn.
Turns per dispatch dropped from roughly six to roughly two and a half. The cheap model's prediction doesn't have to be perfect — it just has to be better than letting the expensive model discover what it needs through trial and error.
One prompt, assembled from parts
We moved from the monolithic prompt (one 8,000-character file that tried to cover everything) to a modular architecture: a small core (identity, guardrails, response style) plus mode-specific modules loaded at dispatch time. Context gating means simpler modes don't waste tokens on data they'll never use. Token count for straightforward queries dropped from 6,800 to 3,100.
One source of truth per module. Assembled at dispatch time. Validated at build time.
The Real Hero: Observability
Every breakthrough above was only possible because we invested in making the system's internals visible. Not logging — structured observability designed to answer specific questions. This was the single biggest factor in finding the right path.
Per-turn executor timing
The single most valuable instrumentation: breaking the executor loop into per-turn metrics (LLM latency, tool execution time, tokens, tool calls, stop reason).
This revealed the 99.7% LLM wait time on sequential tool calls. Before per-turn timing, we were optimising tool execution. After, we knew the tools weren't the problem. That one metric redirected our entire strategy.
Prompt X-Ray
Every dispatch records the exact prompt assembly: modules loaded, context injected, context skipped, prompt hash. When the agent gives a bad response, we look up what it saw. Prompt debugging went from "reproduce and theorise" to "inspect and fix."
World view tracking
A structured tracker records the status, entry count, and freshness of every data source feeding a dispatch. Colour-coded freshness indicators and gap warnings on a dashboard. Before this, an agent could say "no failures" because its test results were stale. Now that the blind spot is visible to both operators and the agent itself, freshness warnings are injected directly into the agent's context.
This is what makes the "move with confidence" interaction model possible. The agent doesn't just have data — it knows how fresh that data is.
The behavioural probe
After our third LLM provider swap went sideways, we built a seven-test behavioural harness:
- Basic tool call — does it work at all?
- Parallel tool calls — does it batch, or go one at a time?
- Tool choice modes — auto, required, none
- Thinking/reasoning interaction — does reasoning suppress tool calls?
- Max tool count — binary search for the breaking point
- Stop reason normalisation
- Latency profile
Ninety seconds to run. Would have caught every provider-related issue we hit. We should have built it first.
What We Learned
The concept proof isn't the hard part. Opus proved god mode was possible. Making it work on production-viable models, at production-viable costs, at production-viable latencies — that's where the real engineering lives. If you can only make your agent work with the most expensive model, you haven't solved the problem yet.
Orchestrate, don't monologue. The biggest leap wasn't making the agent smarter — it was making it a better delegator. An agent that triggers specialised workflows and coordinates their results is faster, cheaper, and more reliable than one that tries to reason through everything from scratch. Build great specialist capabilities first, then let the agent compose them.
Confidence comes from rollback, not caution. Asking the user for permission at every step is safe but slow. Building instant rollback into every mutation changed our interaction model from "may I?" to "here you go." The agent moves fast because undoing is cheap. This only works if you invest in the undo capability first — but that investment pays for itself immediately.
Instrument before you optimise. We spent time optimising tool execution when the bottleneck was LLM round-trips. A black-box metric misled us. Per-turn timing cost a few hours to build and saved weeks of misdirected effort.
Build the probe before you swap. Every LLM provider claims OpenAI compatibility. In practice, they differ on parallel tool calls, parameter handling, reasoning interactions, and stop-reason semantics. A 90-second behavioural test is cheaper than a production incident.
The visibility compounds. Every observability improvement we built for god mode — Prompt X-Ray, World View Tracker, dispatch tracing — also improved our existing workflows. The core pipeline got better visibility as a side effect. Tactical improvements that serve the strategic direction compound; improvements that don't, fragment.
Don't confuse the path with the destination. We went from brute-force multi-step agentic reasoning to delegation-based orchestration with targeted interventions. That might look like scaling back. It's not. The agent does more than the original vision — it does it by coordinating existing strengths rather than trying to be the smartest entity in the room. The destination was always natural-language-driven performance engineering. We found a smarter route.
The agentic AI space is in its "more is more" era. More tools, more reasoning steps, more complex agent graphs. We've been there. We built the brute-force version first. It worked on the biggest model.
Then we found something better: an agent that's fast and cheap because it knows when to act, when to delegate, and when to let the user decide — backed by the observability to tell the difference.
God mode isn't about one model doing everything. It's about the right capability at the right moment, with the confidence to act and the safety net to undo.
Read about how the full agent team works together, or explore our three-layer correlation architecture.