Changing our mind on AI code reviews

Code review is the last refuge of the human software engineer. It feels like agents generate thousands of lines of code per minute, and we are the last line of defense between that slop and production. Letting AI review code would be crazy, right? Who watches the watchmen?

This is the story of how we overcame our fears about AI code review and let our team ship faster than ever.

What happened

In the last 6 months the amount of code being written at Miter has gone up sharply. Each engineer is producing 2-3x more pull requests per day with the help of AI coding agents. Pull requests are also trending larger on average, mostly because AI agents tend towards verbosity (and also add a lot more tests). And the team is growing. More code per person x more people = way more code flying around.

Engineers started to mention that they felt like code review was a bottleneck. We took a look at the stats and we noticed that median time to first review was steadily climbing, and outliers were waiting more than a day to get their code reviewed, which is way slower than we want to move.

This problem is obviously not unique to Miter. Addy Osmani wrote the most succinct explanation that I’ve seen for what is going on:

It was clear we had to do something to address this bottleneck. There is no point in writing a ton of code if we can’t get it reviewed and ship it to customers.

We’re a tech company, and the year is 2026. So the question basically posed itself: “We let AI write code. Should we let AI do our code reviews?”

Skepticism about AI

We’ve had AI code review tools at Miter a long time. Our team has gotten a ton of value out of Cursor Bugbot in particular; it seems to consistently find real bugs or missed edge cases in our pull requests. But there is a big difference between allowing AI to do a first-pass review to find bugs and allowing AI to actually replace humans in code review.

Even if we could magically trust AI to be a perfect code reviewer, we wouldn’t want to eliminate the process completely. Code review is not all about unblocking the author! In a normal code review both the reviewer and the reviewee tend to learn something and grow through the process.

It also wasn’t clear if code review was the real bottleneck or just the apparent one. Code review comes right after writing code, so it makes sense that it’s the first bottleneck you run into when you write more code. But if we removed this bottleneck, would we just slam right into another one? Would making code review faster actually help us ship more, or would we end up redoing work, running into late-breaking issues, getting stuck in QA, etc.

We felt stuck. Clearly we needed to do something, but we also were not ready to abandon a code review process that had gotten us this far. We finally got the courage to try something after reading Charity Majors’ post about this debate:

This didn’t have to be an all-or-nothing debate. We just needed to find a way to ship some code to production without a human reading it, and then we’d have a foundation to build on.

Where to start?

Okay, so we worked up the courage to let AI replace humans in some code reviews. But which ones? Obviously, we didn’t want to just jump into the deep end.

One way to frame this discussion is looking at human “rubber stamping”. There are a lot of trivial code reviews where the human reviewer just hits the approve button after quickly glancing at the code, or maybe not even glancing at it if they trust the author. Our Slack channels had a long history of people asking for a “quick stamp” on small pull requests (PRs) to get unblocked.

If we could turn those rubber stamps into automated reviews, we would remove a source of human toil without adding much additional risk. This cut the scope of the MVP significantly. We didn’t have to build a state of the art AI code review bot, we had to build a system to determine if a pull request meets the conditions for a rubber stamp.

We were able to identify some categories of changes that are pretty much always safe to rubber-stamp:

• Test-only – if an engineer changes test-only code and the tests still pass, we are comfortable letting them ship the PR. We trust their motivations.
• CI-verified changes – simple refactoring PRs (improvements to type safety, abstractions, etc) which change code organization but not functionality. These changes are fully verified by our automated tests. If the code builds and all the tests pass, we’ll rubber-stamp a refactor.
• Documentation – we let engineers edit documentation (internal and external) without code review.
• Internal-only – we’ve got a lot of internal tools (everything from dev scripts to our internal agent) which we allow people to hack on with limited code review. Breaking these tools can’t impact our customers or our critical systems, so we prefer speed to correctness and tend to rubber-stamp PRs in these areas.

We estimated that about 5% of PRs would fit into these buckets. Would that change the world? No, but it would give us proof that we can do something new and allow us to increase our risk tolerance over time.

Why we built, and what we built

At this point we were determined to introduce an AI rubber-stamp bot. The first question we had to answer was build-vs-buy. Surely there must be something out there that we could buy off the shelf. On the other hand, we knew that we wanted a system which is customized to our needs and building didn’t seem too hard.

Right around the time we were having this discussion, Cursor moved from a fixed price to a usage-based pricing model for Bugbot. This increased our bill 10x without adding any extra value. This was the tipping point for us: if we want something we can keep around for a long time we would have to build it.

Unlike previous internal agents we’ve built, we decided to use a framework to get off the ground quickly. We chose the Eve framework from Vercel, which feels like Next.js but for building interactive agents. We have been extremely impressed with Eve so far! It removes almost all of the boilerplate from developing agents. We just focus on our prompts, tools, and logic, and Eve handles all of the wiring, including an excellent local development experience.

With that foundation in place, we built our code review agent around a simple principle: optimize for recall when finding issues, and precision when validating them.

Rather than asking a single agent to both identify and validate a potential problem, we split the review into multiple layers. Each layer is deliberately optimized to over-produce findings, with an independent critic dedicated to validating the findings from that layer. The critic re-derives the evidence from scratch and challenges each finding. If it can’t independently validate an issue, we don’t surface it to the engineer.

With this approach, we have three primary review layers.

Code quality and adherence to patterns

This layer answers: “Is this good code?” It checks the change against a pattern library we’ve hand-curated. This includes a reference doc for each surface (backend, frontend, mobile), rules that apply across the codebase, and a separate pass for general clarity: things like reducing nesting, choosing good names, and adding only high-value comments.

These rules help identify things like violations of established patterns, questionable abstractions, or code that doesn’t follow conventions we’ve explicitly decided on as a team.

These findings are surfaced to the author, but they don’t feed into our risk engine. The goal here is to encode our engineering taste into the AI code review without using code quality as a reason to block a PR.

Change risk

This layer answers: “Is this change risky to merge?” The goal here is to understand the intent and potential risk of the change, rather than simply looking for code that could be improved.

The agent reads the PR description, examines the change, and considers the files and systems being modified. It compares those changes against a list of areas we consider higher risk and reasons about whether the specific change could have meaningful consequences.

This is where we distinguish between a change that is simply large or unusual and a change that is actually dangerous to merge without another engineer looking at it.

System impact

A change can look perfectly reasonable in isolation while still breaking something downstream. This layer looks beyond the context of the current change and traces callers, consumers, and relevant dependencies to understand what else could be affected by the change.

The goal is to catch the class of bugs where diff is correct in isolation, but the code change introduces a bug in other dependent code.

This is the newest of the three layers, so we haven’t tuned it for perfection yet.

Independent critics

Each review layer is paired with an independent critic. This critic’s job isn’t to simply review the findings that the first agent produced. Instead, it attempts to re-derive the findings from scratch. If the agent says: “This change could break X because Y.” The critic independently investigates the code and asks: “Can I find the evidence that this change could actually break X?”

If the critic can’t reach the same conclusion based on the available evidence, we discard the finding rather than showing it to the engineer. That split is doing most of the work: an agent that only has to generate candidates can afford to over-produce, and a critic that only has to verify one claim at a time can afford to be strict.

Turning review findings into a decision

The output from these review layers ultimately feed into our risk engine. The risk engine considers the findings from the different layers and determines whether the PR can be automatically approved or whether it should be escalated to an engineer.

We also maintain an explicit list of sensitive files and areas of the codebase that always require additional scrutiny. This check is deterministic and doesn’t rely on the agent’s judgment. If a PR touches one of these areas, it is escalated to an engineer regardless of how confident the rest of the review pipeline is. This gives us a hard safety boundary around parts of the system where we don’t want an agent making the final call.

The resulting pipeline looks roughly like:

How it’s going

The code review agent has been running for a few months now and things are going even better than we expected.

Let’s start with the vanity stats: in the last 30 days the bot has approved 10-20% of all reviews, which is a major velocity increase for the team. Median time to first approval is down by 50%. This means that we are actually moving more code through the system, we’re unblocking our team to ship more.

Of course none of this matters if the agent is lowering our bar. If we just wanted to ship 10% of PRs without code review we could have build the bot in one line of code:

The really impressive thing is that we have not seen any negative consequences of this rollout. We have not had a single incident or serious bug caused by a bot-approved PR. Our stats in Sentry look like nothing ever happened.

We’re also not seeing an abdication of responsibility by humans. In multiple cases I’ve seen engineers voluntarily get human review on their PRs even after the agent approved them. We still value code review, we just don’t need to do quite as much of it as we did before.

Humans are doing fewer code reviews, which means they have more time to spend on each one. We are finding that the team is taking their code review responsibilities even more seriously, understanding that the things they’re being asked to review are non-trivial.

We’ve also been happy with the decision to build vs. buy. We were able to get something working extremely quickly with the Eve framework. Because we own the entire system, we can customize it a lot more to our needs and train it to look out for the types of issues that are most often flagged in Miter code reviews, something which is hard to do with an external code review bot. And the cost is totally under our control. We’re spending $0.25 per review, vs $1.50 for Cursor BugBot reviews. As new models come out, we’re able to choose the price/performance tradeoff that meets our needs.

That’s not to say everything was perfect. Our bot is very good at identifying which PRs are trivial, but much less good at reviewing non-trivial PRs for serious flaws. We have kept Cursor Bugbot around because it’s still much better than our bot at finding those bugs. We plan to spend a lot of time in the next few months raising the quality of the bot’s deep review capabilities.

What’s next

This experiment has increased our confidence that we can build excellent agents internally that affect our developer velocity. We’re starting to get more ambitious! Next we’re going to take a swing at a full software factory. We want to see how quickly we can go from idea to product, and code review is just one tiny step on that journey.

We’re building the software stack that powers the construction industry, and along the way we’re going to build some pretty cool stuff to ourselves too. And yes, we’re hiring!