Reality Belongs Inside the Loop

"It is really impossible for clients, even those working with software engineers, to specify completely, precisely and correctly the exact requirements of a modern software product without having built and tried some versions of the software they are specifying." — Fred Brooks

Ion Stoica and colleagues at Berkeley have a new paper, Reality Is the Final Verifier (Krentsel, Agarwal, Cemri, Liu, Sankhe, Mao, Zaharia, Stoica; thread). In short:

• Requirement gap: what we write down vs. what we actually want.
• Model gap: the environment we evaluate in vs. the world we deploy into.
• No proof closes either. Following Brian Cantwell Smith (1985), a proof shows the software meets its stated requirements under stated assumptions. It cannot show that the requirements or the assumptions are right.
• Agents widen both. They optimize against the evaluator faster than any human, and they lack the tacit knowledge an experienced engineer brings.
• Two things become scarce: human judgment about what behavior is acceptable, and faithful evaluation under real conditions.

We agree with the diagnosis. SageOx is a ten-person team of humans and AI coworkers. In six months we have shipped a cloud product, a mobile app and a hardware device, about 4.5 million lines of code, and we have fallen into both gaps many times.

We don't claim to have closed them. Dave Griffith describes how our profession has always handled a problem it cannot solve completely: "Fully general verification of software is mathematically impossible per Gödel, but that hasn't stopped us from building partial verifiers, tuned to validate the sorts of software that humans actually wrote." Type checkers, linters and test suites never reached 100%. They reached enough that we could sleep at night.

Everything below should be read the same way. Recorded discussions, walkthroughs, the Ledger, flash-every-step loops and wind tunnels are partial verifiers for intent and for reality. None of them is complete, but each one catches a class of mistake earlier and more cheaply than the last, and together they let a team moving this fast still sleep at night.

Part I: the requirement gap

The swing is a solved problem, for one class of problems

Tree-swing panels by Redrobsche and Belbury, after Mozilla's Fxemoji, CC BY-SA 4.0 via Wikimedia Commons. Our composite uses the same license.

My cofounder Ryan Snodgrass puts it bluntly: if you can write the RFC, the code is done. For a growing class of problems, the gap between specification and implementation has collapsed. What remains is ambition. Most engineers still cut scope out of a reflex formed when implementation was expensive (see Ryan's YAGNI changed altitude). What limits scale now is courage.

The claim holds only for the right class of problem. Dave Snowden's Cynefin framework separates four:

Illustration: Martin Berg, based on Rob England's 3D interpretation of Dave Snowden's Cynefin framework. Found via Harold Jarche.

Clear problems take a recipe. Complicated problems take expertise, but physics does not change its mind, so the spec is writable up front. Complex problems are Rittel's wicked problems: you have to build it to see it, and each attempt changes your understanding of the problem. Chaotic problems demand action first and sense-making later.

The most impressive AI engineering stories of 2026 are complicated problems, attacked with courage and the new Claude and Codex models:

• A multi-cloud CDN at Hugging Face built by one or two engineers (Hugging Face × Google Cloud).
• A single senior engineer at Meta reimplementing an RDS-class system in two months (personal communication).
• Perplexity building a DynamoDB-class store (CobbleDB).

These are translations. The target behavior already exists, so the spec can be written or read off a running system.

Complex problems are wicked. Before SageOx, I founded XetHub. We had seen massive duplication in the data Apple's ML teams stored. We did not know whether that held elsewhere, and no spec could have told us. We had to build it, and the answer came only at Hugging Face scale. It also paid a dividend nobody had specified: smaller shards made content delivery cheaper.

The paper's requirement gap bites hardest in the complex quadrant, where nobody can write the spec on day one.

Control vs. context

Reed Hastings and Erin Meyer, in No Rules Rules, draw a line between two ways of leading:

• Leading with control: direct the work, set KPIs, check progress, gate quality with approvals.
• Leading with context: give people all the information you can so they make good decisions without oversight, and build their judgment for next time.

Most of today's agent practice is control: sharper prompts, fuller specs, stronger evaluators. So are most of the paper's remedies. For complicated problems, that is right. Austin Vernon: "Because computers lack tacit knowledge, instructions must have bit-level detail."

Complex problems do not yield to bit-level detail, because nobody has the detail yet. What works is closer to one senior engineer delegating to another: "Postgres won't work here, figure out something else." That shorthand works only when the two share context.

Chris Argyris's ladder of inference explains why. Every instruction sits at the top of a ladder the speaker climbed alone: from the pool of observable data, through selected data, meanings, assumptions and conclusions, to beliefs and finally actions. Hand someone only the top rung and they can only do what they were told. That is control. Give them the rungs below and they can reach their own conclusions, sometimes better ones. That is context.

The ladder of inference, after Chris Argyris. Diagram by Biogeographist, CC BY-SA 4.0.

At SageOx, every human-human and human-agent conversation is recorded and shared. The lower rungs travel with the goal, so we can hand out high-context goals. Without that record, the only goals anyone can hand out are high-control ones.

A recent example of a wicked problem: murals

We were an AI-native team tired of walls of text, so we had our agents draw cartoons of our discussions. The murals turned out to be a surprisingly high-bandwidth way to stay grounded in our colleagues' work. But no one could have written a PRD for this feature a few months ago. We had to learn two things at once: the sharp limits of what AI agents can draw, and our own human appetite for good storytelling.

Here are four ways the SageOx platform helped us as we built murals.

1. Intent is plural. The requirements doc is where it gets flattened.

What we do: record discussions with speaker attribution, so divergent readings survive the doc instead of dying in it.

2. Intent is only visible at friction.

• Nobody states what they want in the abstract. People react to something concrete. The friction happens in three places:
  • Discussions: humans against each other's framing.
  • Sessions: humans against their agent's draft.
  • Walkthroughs: designers and users against a running build.
• Our gallery of misses shows it. Nobody could have said in the abstract what they wanted from a mural:

I meant a model of team know-how, and a staged rollout across a mesh of servers. The mural heard warfare and drew a fishing net.

What we do: record all three, so the reactions survive. Each correction we made to the mural generator was tacit knowledge made visible for the first time.

3. A correction to an agent is organizational knowledge in its only observable state.

• The paper says agents lack "local domain and organizational knowledge… engineering 'common sense'" but doesn't say where to find it.
• It surfaces when an engineer says no, not like that, we never do X here. That happens dozens of times an hour per engineer. Then the session ends and the correction is gone.

What we do: every session goes into the Ledger, so the tacit becomes searchable instead of evaporating. The team warfare mishearing became a team lexicon that now corrects every transcript in real time. Nobody had filed an issue.

4. Recorded judgment makes the next requirement better.

• This is the everyday payoff. Senior people write better requirements because they were in the room earlier and know where product, business and customer intent meet. A record lets everyone else into that room.

What we do:

• Prime every agent and new hire with what the room decided, so the gap opens narrower.
• Surface missing decisions early. While planning how Knowledge Bubbles belong to teams, Galex Yen's planning agent surfaced an open migration issue and design plan that still assumed team conversations would move into a Knowledge Bubble. Recent discussions had reversed that premise: conversations needed their own store. He brought the team together before implementation, and we closed the migration issue.
• Fluoride in the water. Context arrives unasked. Saving two job descriptions, an agent flagged my week-old decision to go visual-first in marketing:

• Common law, not a constitution. Norms can't be written in advance. Small recorded judgments add up to precedent.

Part II: the model gap

The OxDot is a small, always-present audio device for shared desks, standups and design reviews. It records with one touch, and its Auto Rewind lets a team retroactively capture a breakthrough conversation it forgot to record.

None of us had done RTOS development in decades (how we got into firmware). We had to learn the device while building for it, and our first autonomous agent loop ran into exactly the failure the paper describes with its key-value store: code that passed its tests and did not do the job.

The first twin failed

On March 22, 2026, we received our first hardware device. The first thing we did was create a digital twin from the scant documentation. It gave the agent a place to run, and the loop produced an impressive amount of code. But the twin described the device we thought we had. As it drifted from the real hardware, the code followed. The tests passed for a device that did not exist.

So we put the actual device in every iteration: code → flash → on-device test → parse report → repeat. The loop could still move quickly, but the board now had a vote. A team with no recent embedded experience shipped a working device within weeks. Only then did we build a better twin, calibrated against what the hardware had taught us. Twin, then physics, then a better twin.

A wind tunnel for a microphone

Real hardware caught what no unit test had:

• Silent truncation: the device reported eight minutes of audio. The server had received 2.08 seconds.
• Heap fragmentation: TLS failed against S3, but not our API. S3's certificate chain needed a 16 KB buffer; ours fit in 4 KB. After enough use, the device had free memory but no large enough contiguous block. Days of chaos testing exposed the pattern; we now enforce a minimum contiguous free block of 12,288 bytes.

Other hardware engineers warned us about Wi-Fi hopping problems in this class of device. The documentation and our first twin had not prepared us for that either. We built an end-to-end chaos chain to test how the OxDot handled unreliable connections. Three agents ran it:

1. The singer played an AI voice reading "Mary had a little lamb" and counting one, two, three to the OxDot. We picked the rhyme because Edison recited it on the first sound recording.
2. The gremlin flaked the Wi-Fi underneath it, toggling the access point and simulating dropped connections.
3. The auditor verified that every reconnect happened as expected and that no audio was lost, comparing the transcript against the known text.

We ran it more than 300 times. A new hire alone in the office one evening took it for a ghost.

We spend about 30% of our engineering effort on attestations, fault injection and chaos resilience like this.

Twins are the answer to cost, when anchored

Flashing hardware at every step taught us what the first twin had missed. It also cost time. Digital twins let us run many more tests; they are the primary test substrate for our cloud APIs and TV integrations. A twin earns trust only when real behavior regularly re-anchors it. Where we cannot do that, we use the device itself. Consumer TVs each behave differently, so casting is tested on a bench of real ones.

Reality is the final verifier. It does not have to wait for deployment. Put it inside the inner loop, as a flash step, a wind tunnel or a bench, and it verifies on every iteration.

Part III: the hardest work is ours, and it gets bigger

"To be prepared against surprise is to be trained. To be prepared for surprise is to be educated." — James P. Carse

Part II was training: flash steps, wind tunnels and benches prepared us for surprises we could anticipate. But no test can tell us which problems deserve the team's time, or prepare us for a surprise we have not imagined. That work asks more of people as agents take on more of the execution.

The paper calls human judgment the critical resource. We agree 100%. In our experience, that judgment takes agency, creativity and hard-won wisdom:

• Range: crossing disciplines and choosing to work outside your expertise. I had not touched firmware in twenty years. Knowing which questions physics would ask was enough to start.
• A repertoire of problems: carrying a long list of things worth solving, then recognizing when a model makes one possible and deciding to go after it.
• A feel for the jagged edge: knowing where today's models are superhuman, and where they will confidently build something that only looks right. The wind tunnel was our answer to one of those edges; someone had to imagine it.

The jagged edge moves every quarter, so no textbook captures it. At this rate of change, people at the frontier separate fact from fiction through people they trust. Relationships matter, but the conversations do not make the choice for you.

Alan Frindell showed me where Media-over-QUIC was heading. Safari's new WebTransport support gave me a hunch that real-time presence was finally doable in the browser, on the device and for Oxy. Then Zhenbin Xu showed me Zoom's new speech APIs. They led the ASR benchmarks, but choosing a newly released API over established vendors still took judgment. I chose to put the pieces together and build the live standup. No benchmark could make that product decision for us.

The messy trail

Helmholtz admitted that his published work left out his mistakes. It showed only "the made track by which he may now reach the same heights without difficulty." Most companies do the same. They keep the spec, the design doc and the merged PR, and throw away the arguments, the wrong turns and the twenty drafts that came first. The requirement gap hides in what gets thrown away.

We keep that messy trail. Recording a wrong turn lets the next person, or the next AI coworker, see why we abandoned it and make a better call.

The paper wants evidence from deployment to flow back and revise the requirements. We agree, but we don't wait for deployment. When a designer in a walkthrough says "that's not what I meant," that is evidence from reality too. It just arrives weeks earlier and costs a meeting instead of an incident.

Agents can already write the code. People still have to decide what is worth building, notice when the result misses the point, and change course. That is the hardest work, and the more we can build, the more often we have to do it. The record of those decisions and their reasons has to grow with us.

The point is the people

Ryan recently wrote about why humans and AI coworkers share one context at SageOx: the human connections it makes possible. The requirement gap is the distance between what people mean and what gets built. It narrows when people can see each other's reasoning. The recordings, the walkthroughs, the Ledger and the murals carry that reasoning into the work. In Ryan's words: "Start with the people. Let the agents lift every one of them."

Building SageOx with SageOx has made us faster than I could have imagined six months ago. It has also made our choices matter more. We have to decide what is worth building, help each other see what we mean, and listen when reality tells us we got it wrong. The hardest work is ours, and it gets bigger.