Jon Moshier
Jon Moshier / Notes / Rebuilding Review and Deployment for the AI Era budding
Note · From the Notebook

Rebuilding Review and Deployment for the AI Era

When AI writes most of the code, the human review-and-ship path becomes the constraint. A map of the approaches being tried to fix it: reviewing specs instead of diffs, AI reviewers, stacked diffs, and telemetry-gated deployment.

budding Started July 13, 2026 aiagentscomplexityproducts

If AI generation cuts the cost of writing code but not the cost of trusting it, the human review-and-deploy path becomes the binding constraint. That is the setup established in Code Review Tooling Market - GitHub, GitLab, and the PR Review Bottleneck and AI SDLC Bottleneck. This note maps the responses being tried, and where each one is still speculative. The through-line: the industry is trying to move the trust decision off the individual diff, either upstream to a spec or downstream to runtime.

Move review upstream: review the spec, not the diff

The most structural bet is that reviewing generated code line by line is the wrong unit of work. Spec-driven development inverts the order: write a precise specification first, generate code from it, and review the spec. Spec-Driven Development treats the spec as the source of truth and code as a regenerable artifact.

The mechanism that makes this more than a slogan: if the spec is precise enough (types, contracts, acceptance criteria, formal models), a wrong implementation shows up as a verification failure rather than something a human has to notice by reading. Microsoft’s GitHub Spec Kit formalizes this into a seven-step loop (constitution, specify, clarify, plan, tasks, implement, validate) with the spec as connective tissue across the lifecycle. In safety-critical work, teams are pairing LLM generation with formal verification so generated code must satisfy a model checker before it can ship.

The open question is scope. Writing a spec precise enough to make errors mechanically visible is itself expensive and, for large fuzzy systems, may not be cheaper than review. Whether spec-first generalizes past well-bounded and safety-critical domains is unsettled.

Point AI at the diff

The direct response is to answer AI-generated volume with AI review capacity. This is the fastest-moving segment. CodeRabbit reports over 2 million repositories connected and 13 million PRs processed, working across GitHub, GitLab, Bitbucket, and Azure DevOps. The mechanism split matters: CodeRabbit reviews the diff, while Greptile indexes the whole codebase and reviews each PR against it, aiming at bugs in the seams between files and services that a diff-only view misses.

The benchmarks expose the core tension. On a test of 50 open-source PRs, Greptile caught over 50% more bugs than CodeRabbit, but at a cost of roughly 11 false positives per run. That is the load-bearing problem for AI review: a noisy AI reviewer retrains humans to rubber-stamp, the same skimming the Faros data already measures from volume alone (31.3% more AI-era PRs merging with no meaningful review). A low-precision reviewer is a second, independent path to that outcome. Precision, more than recall, is likely what keeps a human in the loop. Note also the source incentive: these numbers come from a vendor comparing itself favorably, so treat the exact multiplier as directional.

Shrink the unit of review

If large PRs are what break review (LinearB’s data puts best-performing teams under ~225 lines changed), the workflow fix is to make small the default. Stacked Diffs break a large change into a chain of small, dependent PRs that review and merge in sequence, a practice from Phabricator and Google’s Critique now productized by Graphite. Merge Queue handles the integration side, serially testing each approved change against the real post-merge trunk. Both pair with Trunk-Based Development: many small integrations beat few large ones because reviewer attention appears to degrade faster than linearly with diff size.

This is the least speculative bucket. The techniques predate AI and have measured track records. AI changes the emphasis: when generation is cheap, there is no excuse for a large PR, so the small-batch discipline that was optional becomes structural.

Move the safety net downstream to runtime

The deployment-side bet is that if you cannot fully trust code at review time, you make shipping cheap to reverse and catch failures in production fast. The building blocks exist:

The logic is a shift from prevention to fast detection and recovery, the same move DevOps made with mean-time-to-recovery over mean-time-between-failures. The counterargument: runtime guardrails only catch what telemetry can see. A subtle logic error that corrupts data quietly, or a security flaw, will not trip a latency or error-rate alarm. Downstream safety nets relax the review constraint for a class of failures, not all of them.

The accountability question nobody has answered

If AI writes the code and AI reviews it, the human role narrows to approving a policy rather than inspecting a change. That is coherent right up until something breaks and the question is who understood the change well enough to be accountable. The DORA data showing 39% of developers with little or no trust in AI-generated code is not only about quality, it is about ownership. Every approach here relocates the trust decision, none of them removes it. Spec-first pushes it onto whoever wrote the spec, AI review onto whoever tuned and trusts the reviewer, runtime gating onto whoever defined the guardrails. The bottleneck may not disappear so much as change shape into the harder-to-measure work of deciding what “correct” means before anything is generated.

Try it

Measure your AI reviewer’s precision, not its recall (1-2 hours, GitHub + a spreadsheet). Install an AI reviewer (CodeRabbit and Greptile both have free tiers) on a repo and let it comment on the next ~20 PRs. For each comment, tag it true positive, false positive, or nit. Compute the false-positive rate. The research predicts the failure mode is noise: if more than a third of comments are things a developer would dismiss, watch whether reviewers start skimming the AI comments entirely. That skim behavior is the mechanism by which a high-recall, low-precision reviewer quietly reproduces the unreviewed-merge problem it was bought to solve.

Simulate a telemetry-gated rollback (a weekend, any language + a feature-flag library). Wire a trivial service behind a flag (Unleash and Flagsmith are open source), define one health metric (error rate or p95 latency), and script a rollout that auto-reverts the flag when the metric crosses a threshold. Ship a deliberately slow code path behind the flag. Watch the rollback fire on the metric alone. Then ship a silent data-corruption bug that does not move the metric, and watch it sail through. That contrast is the whole argument for what runtime gating can and cannot replace.

See also

Sources

← All notes Read recent essays →