Read the code, drive the browser, stay inside a budget
The goal is not an agent that clicks for ten hours. It is a defensible answer to one question: is this domain ready to ship? That needs code analysis for what should exist, browser testing for whether it works, and a cost ceiling that makes running it routine rather than exceptional.
Why the current design falls short
The agent reads both codebases, derives about sixty behaviours for an entire domain, and verifies each once in a browser. That is a sample — thin, unevenly spread, and expensive.
The first tab a user lands on has no test of any kind. Nothing in the plan navigates there.
A third of every run checks that things hide correctly, while most tabs are never opened to see whether they work.
Every input, every required field, every format rule — two checks between them.
Three and a half hours, producing four findings, one of which was a real regression.
A clipped button on that dashboard went unreported for eight consecutive runs, for two independent reasons: nothing in the plan opens the screen, and the agent never looks at how a screen appears. It captures screenshots, confirms they are valid images, and files them for a human to read.
The same holds for things a QA engineer treats as routine. Console errors are never read. Network calls are never compared between the old app and the new one. Security is not a category the planner can express. And nothing anywhere asks the question a migration most needs answered: what did the old app have that the new one is missing?
Two instruments, two different questions
Reading code and driving a browser are not two ways of doing one job. They answer different questions, and a release decision needs both.
The agent already reads source — that is how it decides what to test. But no conclusion is ever drawn from it, and the two codebases are never compared with each other. Source is an input to planning and nothing more.
Reading the code
- Completeness. Every route, control, permission, validation rule and endpoint — the full set, not a sample.
- What went missing. A screen the old app had and the new one dropped. Invisible to a browser, because nothing is broken — nothing is there.
- What actually changed. So testing concentrates where the code moved.
- Gate-by-gate comparison. Every permission gate against its counterpart in the old app — broader, narrower, dropped or invented. Where five of one day's defects came from.
- Contradictions. A permission the front end uses that the back end never issues. A rule described in a comment but never implemented.
- Cheap and broad. Runs over everything, every time.
Driving the browser
- Emergent behaviour. The clipped button. No line of source predicts it.
- Reality. Code can look correct while the request fails.
- Enforcement, not intent. A gate that hides a button but still serves the data.
- Integration. A button disabled by a computed three files away, obvious on screen.
- Expensive and narrow. Runs where code changed or something looks suspect.
The argument that matters most: the denominator
Today the strongest available claim is "forty-two of sixty-three checks passed." Sixty-three out of what? The number came from a model deriving behaviours until it hit a cap. There is no denominator, so verified is not a measurable claim — which is exactly why nobody can currently be told a domain is ready.
Code analysis produces the real denominator. Then coverage is a fraction, and what was not tested becomes visible instead of invisible.
| Found by reading code | Found by driving the browser |
|---|---|
| A tab gated on a permission that governs a different resource entirely | An archive button that silently did nothing |
| A permission used by the front end that the back end never issues | |
| A button gated on a rule the old app never had | |
| A permission-model divergence affecting most of the product | |
| A comment describing a rule, implemented as though it were one |
The line that keeps this honest
Code analysis may prove absence and declaration. Only observation may prove behaviour.
"The old app has this route and the new one does not" is a fact, and source may assert it. "This code looks correct" is an inference — and inference is precisely how one of today's bugs was born, when someone read a comment describing a rule that had never been implemented, and implemented it. Source may enumerate, flag suspicion and define coverage. It may never issue a pass.
What it costs, and what it must cost
One run of one domain consumed 148.5 million input tokens against 296 thousand output — a ratio of roughly 500 to 1.
That ratio is the whole story. The expense is not the agent thinking; it is re-sending enormous context on every call. Each browser observation returns a full accessibility tree, and it accumulates inside a long session that re-sends all of it, turn after turn.
Which means the obvious version of "test everything exhaustively" is unaffordable. More clicking is more observation is more context. Done the current way, thoroughness and cost rise together.
The reframe: the model decides, code does the looking
Today a model reads a hundred-thousand-token page snapshot to find a button. That is expensive, and it is also worse: a model reading a snapshot notices some of what is there, while a deterministic extractor enumerates all of it and misses nothing.
So most of the exhaustiveness moves out of the model — and gets more thorough on the way.
| Check | Performed by | Cost |
|---|---|---|
| Enumerate every control on a screen | code | free |
| Clipped, overlapping, off-screen, zero-size | code | free |
| Console errors the new app adds | code | free |
| Network calls, old versus new | code | free |
| Routes the old app has and the new lacks | code | free |
| Every permission gate, old versus new | code | free |
| Permissions used but never issued | code | free |
| Validation rules declared | code | free |
| Which surfaces actually changed | code | free |
| Does this control do the right thing | model | paid |
| Does denying a permission block the request | model | paid |
| Does this form reject what it should | model | paid |
| Does this look wrong | model | paid |
Five levers, largest first
Cuts cost and blind spots together. The single biggest lever.
A unit is one panel. Never hand the model a whole page — this attacks the largest context contributor directly.
Long sessions accumulate and re-send everything. A unit run starts clean and finishes in minutes.
A unit that passed at a commit whose source has not moved is not re-run.
Planning is expensive and is currently repeated on every run.
A token budget per unit that fails loudly rather than quietly burning. Target: 1–3M input per unit.
Splitting the work into units
A survey walks the app and writes down its real structure before any testing. Sections that contain other sections keep dividing until reaching a screen that is simply a screen. Each leaf is a unit: its own plan, its own run, its own report.
Why this one change fixes several problems
A unit's plan is small, so it is stable between runs. Today the plan is regenerated whole each time and only about a third survives, so nobody can tell whether a fix worked. Small stable plans make runs comparable — and short runs make iteration a working rhythm rather than an overnight wait.
Choose the login before planning anything
A QA engineer does not write a plan and then hope the data supports it. They log in, see what they have, and plan around it.
Today the agent does the reverse: it freezes a plan, then goes looking for records that fit. Anything the account cannot support is already in the plan and can only be reported as blocked. Three of the last run's blocked cases were exactly this.
Today
Proposed
Selection ranks candidates on more than row counts. Volume — enough records to test a list, a filter, a sort. Usability — the last run tested file handling against a customer whose twelve files all had empty names in the database. Variety — archived and active, more than one type, something to filter between. Configuration — which modules and settings are on, since they decide whether a behaviour exists here at all. Safe to change — which records may be modified and restored.
The output is a capability profile, written before any planning: what this login can exercise and what it cannot. Everything ruled out is declared in the report before a single browser action, with its reason.
The trap
Declaring something out of scope must never become a route to a clean verdict. A gap the profile declares is still a gap: still printed, still counted against "is this verified". It simply stops masquerading as a failure.
What gets checked in every unit
Each unit runs the same checklist and either covers an item or reports why it could not. Green items are computed and cost nothing; red items need a model.
Every control on the screen, enumerated exhaustively from the page itself. Nothing is skipped because nobody thought of it.
What the old app had here that this one does not — the class of migration defect no amount of clicking can find.
Nothing clipped, overlapping, off-screen or invisible-but-clickable. Measured, so it cannot cry wolf.
Errors and warnings the new app logs where the old one did not.
For the same action: the same endpoints in the same shape as the old app. Extra calls, missing calls, changed payloads.
Every gate on this screen against its counterpart in the old app. Broader, narrower, dropped, invented, or guarding a permission the back end never issues — each reported, never silently accepted.
Does each control do what its label says, and does the screen end up where it should?
Required, format, length, boundaries, error copy and placement, recovery after fixing. The rules come from source; the model confirms each fires.
Empty, loading, error, populated — using the ability to hold or fail a request.
Already strong, now capped per unit — and extended to check that denying a permission blocks the request, not merely hides the button.
After a change, does every view of that record agree? Does stale data survive a reload?
Keyboard reachable, focus visible and trapped where it should be, nothing left untranslated.
Visual comparison against the old app for what measurement misses. Advisory only, and always paired with a check that can confirm or refute it.
A hard-won rule about anything the agent sees
While writing this plan, a screenshot showed every filename in a file list reading "file not available". It looked like an obvious bug. The database showed the names were genuinely empty — the app was displaying correct fallback text.
Anything the agent sees must be confirmed against something it can check before being reported. Otherwise a visual reviewer becomes a generator of confident, plausible, wrong bug reports — worse than no reviewer at all.
What we keep, and what we replace
This rebuilds the planning and observation layers. It does not rebuild the machinery that makes results trustworthy.
That machinery was paid for by failures — an early run reported fourteen passes carrying no evidence and nine failures that were not real. Everything on the left exists because of a specific incident.
Keep, unchanged
- Old app as the reference. A fault shared with the old app is not a migration defect. This retired nine false failures on its own.
- Evidence-backed passes. A pass must name a screenshot this run wrote and can read back.
- Trusted permission and module control. The harness changes state; the driver only asks. Every change recorded and verified.
- Fail-closed gates. Anything unproven blocks a release rather than being assumed fine.
- Frozen source, verified binding. The run proves the servers run the code it read.
- Redaction and write custody. Credentials never printed; only data the run owns can be touched.
- Replay tests. Every recorded run replays unchanged — how we know a change did not quietly alter old verdicts.
Replace
- Flat domain-wide planning becomes a survey producing units, each with a denominator.
- Source as planning input only becomes source as a first-class finder of absence and contradiction.
- A model reading page snapshots becomes code enumerating them, with the model reserved for judgement.
- Names invented per run become identities tied to what is tested, so runs compare.
- One long run per domain becomes one short run per unit, composed into a domain report.
- Unbounded cost becomes an enforced budget per unit.
- Judging from descriptions becomes judging from recorded evidence — a mistake already found twice.
How you will actually use it
Planning and running become separate commands. Today they are one process, so re-checking a single screen means re-planning an entire domain — which is why iteration is impossible, on the product and on the tool itself.
Two commands
Plan — chooses the login, surveys the domain, computes the denominator, writes a durable breakdown. Slow; happens rarely.
Run — takes one unit and tests it. Minutes; happens constantly. Also how we iterate while building the tool.
Richest suitable account, with a written capability profile.
Units, denominators, and what changed since the old app.
Completeness, layout, console, network — computed, no model.
The model exercises what needs judgement. Minutes.
Passed, broke, and what the data could not support.
In the app, or in the agent when the finding is wrong.
Same plan, same names — so the difference is real.
Freshness, so a verdict cannot be assembled from stale parts
Units run over several days would otherwise compose a report from results at different versions of the code, some of which no longer exists. Every result records the version it ran against; the domain view shows age beside verdict; anything predating the current version is marked stale and excluded from verified until re-run. Re-running a unit is minutes, so staleness stays a prompt rather than a problem — and it doubles as the rule that keeps cost down.
Build order
Sequenced so each phase is useful alone and nothing rests on something unproven. Every phase ends with a real run and a measured token cost.
- Split planning from running. Two commands. Everything after is built and tested through the second, so the build loop is minutes rather than hours.
- A durable work breakdown on disk, readable and correctable by a person, regenerated only deliberately.
- Identity that survives re-planning. Two runs of the same unit must produce the same names.
- A token budget per unit, enforced, failing loudly. Report tokens per run alongside results from here on.
- Freshness stamps on every result.
- Completeness diff between the two codebases: routes, controls, permissions, validation rules, endpoints. Anything the old app has and the new one lacks is a finding on its own.
- Permission-gate parity. Every gate in the new app against its counterpart in the old one, classified: broader (allows where the old app denied), narrower (denies where it allowed), dropped, invented, or no counterpart because the surface is new. Cross-checked against the back end so a gate guarding a permission that is never issued is caught. Broader is a security finding; narrower is a behaviour regression; both get reported rather than resolved by the agent.
- Declared versus actual. Where the old app's written rule and its running behaviour disagree, report both and pick neither silently — a stale comment implemented as though it were a rule is how one of today's regressions was created.
- Contradiction checks — a permission the front end uses that the back end never issues; a rule described but not implemented.
- Change detection, so later browser effort concentrates where the code moved.
- Deliver this before any browser work. It is cheap, and it may find more than the last several runs combined.
- Invert the order — selection currently runs after the plan is frozen.
- Rank on usability, not row count.
- Emit a capability profile before planning, declaring every gap up front.
- Exclude the agent's own leftovers from selection, now that cleanup is off by default.
- Discover structure from routing and components, then confirm it in the running app as the selected account. Anything found in only one is reported, never assumed.
- Emit a unit tree with denominators — for each unit, the complete list of what exists there.
- Containers versus units. Settings is a container; its children are units.
- Deterministic extraction of every control, feeding the model a list rather than a page.
- Scoped observation — the model sees the unit's panel, never the whole document.
- The checklist, with permissions capped so they cannot dominate.
- Run against the smallest real unit and iterate until the report is trustworthy and inside 1–3M tokens.
- Layout measurement first — free, and cannot produce false alarms. Acceptance: it catches the clipped button that eight runs missed.
- Console capture and network recording, both compared against the old app.
- Hold and break requests, unblocking loading and error states.
- Download inspection, then visual comparison last — the only one that can be wrong, so it arrives once everything around it is solid.
- Run every unit, one at a time, within the domain budget.
- Compose a domain report with a coverage grid against the known denominator.
- Prove generality on a second domain, which has never been attempted.
- Hidden-surface reachability — if a control is hidden, is the screen still reachable directly?
- Enforcement, not concealment — does denying a permission stop the request?
- Bounds — does a record outside the current location or account stay out of reach?
Handoff
Before trusting anything in this section
Everything here was true on 27 July 2026 and nothing updates itself. Start by running git log --oneline -20 and npm test, and read docs/STATE-2026-07-27.md in the repository. Where a document and the code disagree, the code is right. Older documents in docs/ describe a harness that has since changed and are marked superseded.
Ground rules
Known traps
- Case identity is unstable across runs — two consecutive runs shared only about a third of their case names. Fix first, or you cannot measure your own work.
- Screenshots are new-app-only. The last run captured 84 of the candidate and 2 of the reference. Old-versus-new visual work needs both lanes first.
- Record selection ranks by row count, not usability — which is how file tests ran against a customer whose files all had empty names.
- Selection runs after planning. Backwards; Phase 3 inverts it.
- Cleanup is opt-in and off by default; the database is re-seeded instead. Selection has no filter excluding the agent's own leftovers — add one.
- Driver variance is unexplained. Two runs differing only in a permissive guard produced 24 and 10 passes. Never treat one good run as proof.
- Typecheck misses template identifiers in single-file components; only tests catch them.
- The last run's stored report understates it badly — 16 passes recorded where the true figure was about 42, because a gate discarded valid results. Stored numbers do not self-correct.
Definition of done, per phase
- Phase 1 — a stored plan produced once, a named unit run against it repeatedly without re-planning; the same unit planned twice yields identical identities; a result stamped with an older version reports stale; every run prints its token cost.
- Phase 2 — a completeness report listing what the old app has that the new one does not, plus a gate-by-gate permission comparison classifying every difference as broader, narrower, dropped, invented or new-surface — produced with no browser session, for well under a million tokens.
- Phase 3 — a capability profile written before any plan exists, and a run where every declared gap was stated up front rather than discovered as a blocked case.
- Phase 4 — a durable unit tree with per-unit denominators, containers and units distinguished, every source-versus-live discrepancy reported.
- Phase 5 — one unit fully covered, every control accounted for, each unchecked item naming why, inside 1–3M input tokens.
- Phase 6 — each sense lands with a test proving it detects a real defect and stays quiet on a healthy screen.
- Phase 7 — a coverage grid against a known denominator, and one unit of a second domain.
- Phase 8 — a security check that demonstrably catches a surface reachable directly while its control is hidden.
Decisions that are not the agent's to make
- A permission-model divergence between the two apps affects most of the product and is documented in the repository. It needs a product decision before further gate work.
- One tab's permission rule changed in a merged fix, following the old app's declared contract rather than its actual behaviour. Deliberate, but a live behaviour change.
- How exhaustive is exhaustive. Every control on every unit is achievable but trades against run time and cost. That ceiling belongs to whoever reads the reports.