We built an AI factory at BetterEngineer in which code makes every decision about where work goes and agents do the work, then ran real tickets from Linear through it and fixed the handoffs between its stages as they failed.
Written by Josh Arnold , CTO, and Germán Valencia, Lead Architect at BetterEngineer
_______
Developers are starting to hit a new kind of bottleneck: “how do we safely get AI written code into production?” Once coding agents can produce reasonable patches, the scarce resource stops being tokens and model capacity and becomes senior engineers’ attention. In Faros AI’s study of more than 10,000 developers, teams with high AI adoption merged 98% more pull requests, but their review time rose 91%.(Source: The AI Productivity Paradox Report, 2025)
At BetterEngineer, we hit that wall ourselves. An agent can take a well-scoped ticket to a pull request in under an hour, but reviewing that pull request properly still takes one of our senior engineers much longer, because someone has to read the code carefully, catch what the agent got wrong and send it back for another try.
So we built an AI factory, a system that takes a ticket from our backlog in Linear through triage, planning, building and verification before handing it to a person. Our initial goal is for about 30 percent of our tickets to go through the factory from start to finish, and to hold every one of them to a deliberately high bar, a pull request so g that the reviewer's only job is to click "Merge."
In this article, we will describe how we designed that factory, how graph engineering keeps it observable and safe, and what we learned by running real production tickets through it.
Every agent‑opened pull request still needed a senior engineer to read it line by line, check it against our standards, and decide whether it was safe to merge. That review load concentrated on the same small group of experienced engineers, and quickly became the new constraint.
Part of that review is mechanical: whether the code compiles, whether the tests pass, and whether the change copies a block of code that already exists elsewhere. The rest takes judgment about whether a change follows the patterns our team has agreed on.
As Simon Willison (co-creator of Django) put it, "Almost anyone can prompt an LLM to generate a thousand-line patch and submit it for code review. What's valuable is contributing code that is proven to work."
Our goal with the factory was to push as much of both as possible into the system itself. We wanted every factory pull request to arrive having already been treated the way a careful human reviewer would treat it, so that, in most cases, the reviewer is validating a strong candidate for merge rather than doing the entire review from scratch.
We started by thinking about agents, meaning which specialist reviewers we needed and what each one should look for. We made more progress once we looked instead at the path a ticket already takes at BetterEngineer. It moves through a sequence of stages, each with a clear owner and a clear handoff, and the design question became which of those stages machines could own and how work should move between them.
The AI engineering community has recently started calling this graph engineering, which means designing a system as separate stages joined by explicit routes, where each stage can be an agent, an ordinary program or a person. The name is new, but the idea isn’t. It’s the principle behind the deployment pipelines that DORA’s research links to high-performing teams, where every change takes the same automated path to production.
The part of that idea we care most about is who makes the decisions. In our factory, every decision about where a ticket goes next, and what it has to pass to get there, is made by ordinary code that we wrote and tested. That code often hands work to an agent, but the handoff itself is deterministic and happens the same way every time. Agents do the work inside a stage, and they never choose the route.
Our factory picks up tickets from Linear and follows the path a ticket already takes. An agent triages it and writes a plan, a person approves the plan, another agent builds it, and the result goes through verification before a human reviews the pull request.
Verification does most of the checking, and it is a small graph of its own. It starts with twelve mechanical checks covering type checking, linting, duplicated code, dependency rules and the full test suite. Germán, one of the authors of this post, had just consolidated these into a single command for our CI pipeline, and we had the factory call that exact command, so code written by an agent is held to the same bar as code written by any of us.
These mechanical gates are deterministic. They pass or fail a change the same way every time, and no agent has a say in whether a change gets past them.
That matters more as AI writes more of the code. DORA's 2025 research found that AI adoption now goes hand in hand with higher delivery throughput, but also with rising instability. In DORA's words, AI acts as "an amplifier" of whatever process it lands in.
If the checks pass, four specialist review agents read the change in turn, one each for API routes, database access, naming, and consistency with patterns already in the codebase. A router, which is plain code, decides which reviewers see which files based on the paths a change touches. Review stops at the first confirmed problem, and a problem only counts as confirmed after a fifth agent has tried to prove it is a false alarm and failed.
A failing result sends the work back to the build agent with a short brief, and the fixed version comes back through the same checks. After three failed rounds, the factory stops and leaves the work for a person.
Stages and routes in the factory. Agents do the work inside the blue stages. Every route between stages is decided by code or by a person, and the verification loop stops after three rounds.
The same principle governs what happens after verification. Each reviewer returns a verdict in a fixed format, and a short piece of tested code reads it and routes the work. A pass goes to human review, a failure goes back to building with the reviewer's findings, and a third failure goes to a person. The build agent can say it has finished, but code decides where its work can go from there, and nothing reaches human review without a passing verdict.
Every move is logged with the time and the rule that caused it, and when something went wrong, that log usually showed us why within a few minutes.
We switched on the verification stage as soon as Germán's consolidated gates landed on our main branch, and started pushing real tickets from our backlog in Linear through the factory. One of them, a change to keep a company's billing email in sync with its Stripe customer, became our main test case and took three attempts.
The first ticket made it all the way through verification and was merged after a small manual fix. The first run of the billing change was less straightforward. It lost work in a handoff between stages, which is when we added a watcher that restarts verification runs that go quiet.
We then discarded that attempt and filed the ticket again. This time, we watched the factory close the new ticket as a duplicate of the one we had just deleted. We fixed that behaviour, ran it again, discarded that result as well, and filed a third version.
The third attempt went through triage, planning, building, and three rounds of verification before opening its pull request. The factory then made a change we had requested on that pull request, verified it again, and we merged it.
Across those iterations, the factory wrote all four commits in that pull request, including the change we asked for in review. Verification caught a block of duplicated code and a missing authorization check before any of us read the code. Apart from approving the plan, a person requested a single change in review and then clicked “Merge.”
That is the outcome we want for every ticket the factory takes on, and we reached it by iterating on two things at once: the ticket and the factory itself. Over the same period, two other factory pull requests, a fix for noisy error reports and an authorization fix for client feedback, were also merged without a line of human-written code, while two more were merged after a small manual edit.
Most of the problems we hit happened between stages. The agents generally did what they were asked; the brittle parts were the handoffs. The good news was that those handoffs were implemented in code, so every failure had a specific cause we could find and fix.
We saw three main classes of issues.
In two cases, we lost work outright:
We fixed these by making each run complete cleanly before the next starts, and by treating “empty” reads as a signal that something is wrong, not as a success.
In two other cases, the model misjudged its place in the graph:
Our fix was to disable copies in this context and to route review comments back through the build stage, where the only job is “make this change.”
The surrounding tools created their own edge cases:
Even our safety net had a bug. The watcher that restarts any verification run that goes quiet for three minutes had a counter that rarely reset, so a card that stalled twice in one evening sat untouched for almost twelve hours. Germán noticed the pattern before our monitoring did, which led to several of these fixes. Once we corrected the counter, the watcher’s first check the next morning found the stalled card and restarted it.
When a run went wrong, we usually deleted the ticket and filed it again instead of enriching the same card and trying once more. That can look wasteful, but adding detail in place had real costs.
In our factory, each ticket carries one long conversation with the agents that work on it. Plans we had rejected and code we no longer wanted stay in that history. New detail is added on top of old attempts, and agents keep reasoning from everything that came before. Some decisions also stick to a card: once triage has classified a ticket, that classification cannot be changed. When triage labelled one attempt as a duplicate, no amount of extra detail could move it forward.
At the same time, we were changing the factory at least as much as the ticket. Refilling the same request after each fix gave us a clean test of whether the fix had worked, and because the factory does the building and checking, each discarded attempt cost us very little.
We do not want refiling to become a habit, so we changed the factory to make it less necessary. When triage now finds a closed, unmerged pull request for the same problem, it ignores that pull request’s code but reads the human review comments and passes each objection to the planner as a constraint. A comment one of us had left on an earlier billing attempt, asking that the Stripe update not live in the request handler, shaped the design of the version we merged.
Feedback on an open pull request now goes back through building and verification, which is how the final change to the billing pull request was made without filing anything new. The case we have not solved yet is a ticket that triage misclassifies; that one still has to be filed again.
The choice we would defend hardest is keeping every routing decision in code. In the terms of Anthropic's guide to building effective agents, the factory is a workflow, not an agent: models and tools are orchestrated through predefined code paths, which trades some flexibility for the predictability and consistency we needed.
A model can write an excellent verdict, but if a model also decides what happens next, nobody can say why a piece of work ended up where it did. Twice an agent misjudged its own role, and both times the routing code kept the work where it was. Deterministic routing was also what made the factory debuggable, because every move could be traced to a clear rule. It’s the same reason observability practice puts so much weight on systems that explain themselves from the outside: when every transition is explicit and logged, you can ask why something happened without reconstructing it from guesswork.
Running the mechanical gates before any review agent paid off as well. They answer most questions in a few minutes, the same way every time, and leave the review agents to spend their time on judgment instead of mechanics.
Each loop in the factory also has a cap that ends with a person. The fix loop stops after three rounds and the watcher after three restarts, so no work circles indefinitely without someone seeing it.
Once we settled on our stages, they barely changed. Almost all of the work after that went into making each handoff happen reliably every time. If we were starting again, we would build the stall watcher before the first agent.
We are extending the stall watcher to every stage of the factory, and we are tracking two numbers: the share of our tickets that go through the factory, and how many of those come out as pull requests we only need to merge. Together they tell us how close we are to the 30 percent goal.
Coding agents are getting better quickly. According to Stanford’s 2026 AI Index, top models’ scores on SWE-bench Verified, a benchmark built from real GitHub issues, went from about 60% to near the human baseline in a single year. We expect more engineering teams to find that writing the code stops being the slow part. When that happens, most of the engineering goes into how agents, checks and people pass work to each other. If you are building an AI factory of your own and want to compare notes on graph engineering, we would be glad to hear from you.