How I verify agent-written changes
A walkthrough of Koi Katana’s browser scenarios, replay checks and evidence manifests, and how they support review of AI-assisted changes.
Coding agents are useful in my development process, but a finished diff is only part of a finished change. I also need to understand what happens when someone uses the feature, which checks cover it, and where uncertainty remains.
In Koi Katana, my TypeScript and Three.js game, I built a verification workflow that makes those questions easier to answer. This is a walkthrough of the checks in the project. It does not describe a confidential work incident or present the checkout example on my homepage as a real failure report.
Start with something observable
A task needs a behaviour that can be checked. For a fishing change, that might be a player casting, hooking a fish and finding the catch in their inventory. For co-op, it might be players joining the same room and seeing shared state.
The scope matters. A screenshot of a menu does not establish that the interaction behind it works, and a passing fishing scenario says nothing about multiplayer reconnection.
I use feature-specific browser scenarios to exercise those paths in the running game. A phone-sized viewport keeps the controls and layout close to the conditions the game is designed for.
Give each check a job
The checks operate at different levels:
| Check | What it helps establish |
|---|---|
| Seeded combat replays | The same inputs produce reproducible outcomes |
| Protocol and save tests | Network inputs and older save formats are handled as expected |
| Co-op browser scenarios | Players can complete an interaction in the running game |
| Screenshots and recordings | The behaviour can be inspected after the run |
| Entry chunk budget | Startup JavaScript stays within a defined size constraint |
These checks complement each other. A simulator can expose a rules regression without rendering a frame. A browser can show that an interface is usable while missing a rarely exercised rule.
Evidence must match the change
One detail I care about is whether a recording still applies to the code being reviewed.
The browser harness writes a run manifest with the commit, scenario, viewport, outcome and artifacts. Artifacts have hashes so the evidence check can detect a missing or changed file. The checker also rejects failed runs and code revisions that invalidate the recorded evidence.
That avoids an easy mistake: recording a passing scenario, changing the implementation, and continuing to cite the earlier run as proof.
A report should make the limits visible too. The evidence tooling has a “Not driven” field for paths that were not exercised. Naming a gap makes it possible to decide whether another check is needed.
A concrete asynchronous boundary
Co-op is lazy-loaded so that every player does not need its code during startup. That creates an asynchronous boundary worth testing.
The loader tests cover a module arriving after the loading deadline, a player leaving before it arrives, and an older request resolving after a newer request has started. Those paths need different outcomes: a late result can still be useful, but it should not reopen an abandoned interaction or replace a newer request.
This is the kind of detail I want visible during review. The question is more specific than whether a loading indicator appeared: does the eventual result still belong to the player's current action?
Be precise about performance claims
The build enforces a 200 kB gzipped ceiling for the initial JavaScript entry chunk. Three.js and lazy-loaded features are separate chunks. Images, styles and other assets are outside that measurement.
A pull request that grows the entry chunk by more than 1 kB needs an explicit reason in its description. The fixed ceiling still applies. This turns growth into a visible decision while keeping the measurement's scope clear.
Review still requires judgment
A passing scenario tells me something about the path it exercised. It does not prove that the architecture is appropriate, every failure is handled, or the change is easy to maintain.
I use the evidence alongside review of the implementation and its tradeoffs. If the code changes again, the relevant verification may need to run again. If the scenario is too narrow, more evidence is needed.
That is how I want AI to improve my work: help with implementation, then leave enough evidence that I can make an informed decision about what ships.