How we've evolved our CI/CD in the agentic era

Embracing agentic-era development has meant rebuilding our entire CI/CD, deployment and PR pipeline at SageOx, more than once.

As you can see from the graph here, the pace of PRs per engineer is high. And I'll say, it's much, much higher for us than even this graph shows if you look at how we were operating before, because we tended to put a lot of stuff in every PR. A thousand-line PR is small for us in a lot of cases. We would have 10,000-line PRs frequently, and 30,000 or 50,000 was not crazy, which in the old world would have been crazy. So the PR count per engineer is often even misleading in this world.

Massive PRs

Part of the reason we bundled up so much: before, we were still doing git bisect. That's a technique that doesn't work as well in the agentic era for humans who are trying to do manual reading of the output, or the volume at which you have to do it is really high, so it's hard for a human. We've of course moved beyond that, and now we have our agents bisect, which makes things a lot easier. But when we first started doing agentic stuff, we were still embracing some of these techniques we'd been using for decades. So that's basically why we shoved everything into these massive PRs with a short description of what the hell changed. Let the PR carry the burden of the description. The merge itself was still pretty lightweight, too.

And what we found is that we were just waiting.

We also added a lot of tests. Tests can be a problem, so you have to kind of beat them back, but we found them exceptionally useful, even a large number of them, because our surface area is large. Our biggest thing is how we do user-level testing. We still haven't released our open source test package yet, but it's coming, and it's a big deal for agent development. It's hard to describe how big of a shift and up-level that makes for a team delivering product features. Not if you're just delivering small little features, or small things buried within infrastructure.

Version five

We've iterated on this quite a bit. This is, I don't know, version five of our process, let's say, and I'm sure there'll be many versions to come in the coming weeks or months as we adopt other tools that make sense for us.

We've talked about it before, but one of the things we have is anti-entropy around our systems. Agents are looking at bugs, customer issues and other things 24/7 and filing issues, and then other agents are actually implementing those and handing off PRs for either human or agentic merging, depending on the scope. That's a powerful feature of this whole system. I don't know if it's required for what we're doing, but it certainly makes this kind of approach a lot more palatable.

Before

Before, we used to have large CI/CD runs, and it could take twenty minutes end to end to run through all our CI/CD, maybe longer sometimes. That was months ago. So the first step was optimizing that. Like, holy cow, what can we be running on? What are the right AWS instances for our build system?

We hit a wall pretty quickly. Amazon's AWS is not like it used to be, where you just dial up and down and consume what you want. There are so many gates and checks and balances in the system around capacity, and you have to get sign-off from sales engineers on certain tweaks and changes, which is really a non-starter. We don't talk to sales engineers. Sorry. That's just not part of our flow, and we don't do capacity planning way out on these projects. They're one-day projects, not "okay, this quarter we're going to go tackle CI/CD." So we were maxing out the number of builds we could do.

The first thing to bring it down was looking at what we could parallelize in the CI/CD, so we could spin up more onto parallel runners. We sharded tons of our tests, and tons of the things we were running to validate database migrations. Everything. That was a big win, and it brought the times we were waiting way down. Of course, what got annoying is you look at CI/CD and there are 30 or 40 things running, because so many things are sharded six ways or whatever. So that was step one, and it did make some improvements.

Code review limits

Then we were hitting code review limits because of the volume of PRs, so we pushed code reviews back into a first round locally. Beyond what the developer runs, we added Makefile support to call out to CodeRabbit. That helped a bit, because a lot of things were getting caught before the PRs. The downside, maybe even worse, is it slowed down everybody's process on their individual machines. I tend to have at least five to 10 completely independent agentic sessions going, and each session has many subagents, of course, so all of those are paying for those long runs. That's something we're still iterating on: what's the right level before our PRs are created? That will be the next thing we tackle.

Merge queues

So what did we do with our PRs? The first thing is we moved to merge queues. We were creating PRs at a rate where just trying to keep things ready to be merged was impossible. Trunk.io's merge queues have been fantastic. There's no friction for the developer, really, that we've noticed. In fact, it's just much more delightful to work in our repository, because once Trunk has it scheduled, it's just going to go once everything is green.

You're not waiting around. You can say, okay, I'm done with the session. A big part of the goal was how quickly you can get these sessions, these tabs and coding agents cleared off the developer's machine, so we can give them back some amount of human attention. Just having that tab sit around is a drag on human attention. That was the first step, and it made a big difference. Just having merge queues with PRs, forget about any of the CI/CD.

Blacksmith

The next thing we observed was that we were rerunning these really expensive end-to-end tests many, many times, every few minutes, certainly every hour. First off, that was costing us a lot in GitHub Actions runners. We're using Blacksmith now, which is fantastic. Love Blacksmith.

I should have noted this earlier. We moved not only to rightsizing the machines, but also picked a GitHub Actions runner system that had very, very fast options. One of the really great things was using their embedded AI to do an analysis on our end-to-end CI. It gave a lot of suggestions along the way, which achieved so much before we really blew things up and changed the way we worked. If you're on GitHub Actions, I highly, highly recommend you move to Blacksmith and then have it optimize your runners and CI/CD. It picked up little nuances and insights that we didn't by ourselves, analyzing our own development process. That was a big deal.

Two stages

So great. Now we have high-performance action runners and merge queues, and we've split our tests and CI/CD work between two stages: one on the PR, and the second from the merge queue. The nice thing about the merge queue is it bundles up some number of changes, either over a predefined period of time or by number of changes, so when it hits that amount it creates a merge.

That was a big deal, because now the PRs finished faster for developers. They could see it's all green. Everything's done, all the tests are done, code reviews are in flight or done, so I can close my tab. I'm done. I know eventually it'll get out.

It's really the time from a developer's commit push to when they can close their tab that I think matters. When they can easily say, hey, this is taken care of, I don't have to worry or spend money. That's what we're optimizing for in this whole process, and there's still a lot left on the table ahead of us.

Take the slow tests out of the merge

Okay. This is a great situation. Everything's a little bit faster, and people are delighted. Weeks go by, and then people start complaining. Why is it slow? It's slow because we still have some stuff that could be moved around and shuffled.

One of the realizations was that the real goal is from a developer's commit push, when they're done with their feature, to having it show up on test as soon as possible. That's our test or staging environment, essentially, where people can play around, experiment, and see how it works with the features they're developing at a rapid pace. And right now we're feeding it through all these really, really expensive behavior-driven development (BDD) tests and end-to-end tests. You need to do all this stuff just to get it out.

We've been doing releases to production manually. I'll talk more about our release process another time, since we're moving to a less manual, kind of interesting strategy for still getting things to customers quickly while reducing a lot of risk. But since we cut releases manually, as long as we're keeping main buildable generally, I'm actually willing to give that up a little bit for speed.

So I was like, well, why don't we just run them on a periodic basis? We'll run these really expensive BDD and deeper tests off of main, out of bounds of the merge. Take them out of the merge and say there's going to be stuff that merges to main that maybe breaks some sort of test, but it's rare. That burn is relatively rare, and there are other checks and balances. We still want them all to run, so let's just make them run on a cadence, whether it's every four hours, every two hours or every hour. It doesn't really matter. The point is we don't need to run them for every merge.

So we switched to that. We run these and then file any tickets we find against the recent merges in that area to resolve. That could be a human or an agent, it doesn't matter. If it's appropriate, we also tag our coding agent. The way we have it set up, as I mentioned before, is we have 24/7 agents watching certain labels and picking up tasks that are low-risk fixes. So in many cases I can just tag in a non-human agent, and it'll fix whatever the bug was, because maybe it's not very intrusive, it's very minor, and then commit back to main. This, I think, is yet again another really massive improvement for keeping that flow high.

Optimizing for different things

You're optimizing for different things now than you were pre-2026. If you're embracing agentic-era speeds, your whole process must change. You want to be able to see and experience these features coming together at a much, much faster rate. Things that would have taken months and months and months to happen now happen within a day, or days, and that means all the other people on your team are moving at that speed too. These things come together really, really quickly. You want to see that and feel it, especially since you're maybe not completely aligned all the time with all your peers. You need to see it, test it and experience it, and that's also why BDDs are important.

That's a rough high level of where we're at today, October 2026, in how we manage our process from developer to at least our test environments.