Self-Improvement in Production Isn't One Flywheel

Every production agent platform has a number its team does not enjoy looking at: the share of user queries that fall through to a fallback because no agent can handle them. The industry has many names for it. Amazon Lex counts each one as a missed utterance. Microsoft’s Copilot Studio reports “unrecognized” and “unanswered” questions. Google’s Gemini Enterprise publishes the opposite number, a successful answer rate, and the gap below 100% is the same thing. The intent-classification literature calls it out-of-scope.

Most teams work on this number. What bothers me is how little they get out of it. Every failed query is the most concrete demand signal a product can get: a user telling you, in their own words, which capability is missing. If you treat that as a percentage to push down, it is a health metric. If you treat it as a backlog of things to build, it is a roadmap.

I think the next real step for agent platforms is to treat each of those failures as input to a build pipeline. Something specific and certifiable: bank the failure, mine the pattern, build the missing capability with a coding harness working alongside a human, certify it the way you would certify a human-built one, and redeploy. That end-to-end discipline is what I mean by RSI agentic system design: recursive self-improvement run as an operational architecture.

Teams already try to close these gaps by hand. Someone exports the fallback logs, pastes a sample into a chat assistant, and asks for ideas. In my experience that works once, and then it drifts. An assistant sitting outside the system sees snippets. It does not see the full traces, the registry of what already exists, the golden sets that define “correct,” or the certification results from the last ten builds. Decisions made without that context are guesses. The design in this post keeps the builder inside the system, where the context lives. That one choice is, I believe, what makes self-improvement compound instead of drift.

Nothing here requires a research breakthrough. Every stage exists today, shipping in some product or described in some paper. What I would like to describe in this blog is the wiring: connecting those stages into certified, self-closing cycles and measuring them the way you would measure a product.

First, the word the rest of this series leans on: an axis. An axis is one independent dimension along which a deployed platform can improve itself. It is a distinct kind of change the system can make to its own makeup, with its own way of failing, its own way of measuring “better,” and its own way of shipping the change safely. The axes are independent the way a product’s health dimensions are independent: movement along one does not automatically move the others. Which capabilities exist is one such dimension: the roster of things the platform can do at all, which grows when unmet demand is found and a missing capability gets built. How the growing pool of capabilities is organized is another: which agent fronts which kind of request, who hands off to whom, and whether routing stays sharp as the roster expands. Whether the tools those capabilities depend on still work is a third: the APIs, schemas, and integrations every agent’s hands are made of, which age and break on their own schedule. I keep them separate because each one runs on a different failure signal, builds a different kind of artifact, and needs a different definition of “safe to ship.” A certification gauntlet designed for a brand-new agent will not catch a bad reorganization, and a metric that celebrates new capabilities will never notice rotting tools. The separation is also what makes recursion tolerable in production. Improvement confined to one axis at a time, with its own gates and its own metric, stays bounded and auditable. Improvement smeared across every dimension at once is the vague flywheel this post argues against.

This post is the first of a multi-part series. Part 1 lays out the shared skeleton, then walks the founding axis in depth: Coverage, where unknown queries become new certified agents. Part 2 walks the two axes that run in parallel once coverage is live: Structure (the counter-force that reorganizes the growing pool) and Tool Foundry (the axis that repairs and extends the system’s hands).There are more axes I plan to add as well. Coverage ships first, and everything else builds on it.

One skeleton, many axes

Here is the pattern of a production self-improvement story I think is reliable based on my understanding of the field -

production signal → bank → offline mining → automated build (a coding harness, steered by human domain input) → certification gates → human approval → staged redeploy → replay & close → measured KPI → repeat

Recent surveys of self-evolving agents slice this space by what evolves (weights, prompts, memory, scaffolding) or by training stage (Self-Evolving Agents, Self-Improvements in Agentic Systems, and July’s RSI survey). Those surveys answer the question “where inside an agent can improvement live?” This article answers a different question: “what signal triggers a change, and what proves the change did no harm?” Same field, two indexes. If you slice the space by this question, which production signal feeds the cycle, what artifact gets built, which gates make deploying it safe, and which KPI tells you it is working, and self-improvement stops looking like one flywheel. It comes apart into a family of certified closed loops, each axis with its own signal, its own bank, its own build action, its own gate set, and its own north-star metric.

None of this is a new learning algorithm. Control theorists have run monitor-analyze-plan-execute cycles for two decades; the closest published relative is the MAPE control-loop discipline, and this post applies that discipline to capability growth. If you want an ML anchor it would be similar to a coordinate ascent on system quality. Each axis improves one coordinate at a time, and certification decides whether a step is accepted.

The three axes are complementary functions of one system: coverage creates new capability, tool foundry equips it with reliable hands, and structure reorganizes what growth accumulates. A platform needs all three for the same reason a growing team needs hiring, tooling, and the occasional reorg: each one, run alone, eventually creates the problem another one solves. The same skeleton admits more axes which I plan to elaborate on in detail in the upcoming parts.

The system, zone by zone

Every axis in this series runs on the same five-zone architecture. Let me name the zones once, as a map, and walk them in the next section.

• Zone A, production serving: This zone is the live system users touch. Whatever an axis treats as its failure signal (a query nobody could handle, a misroute, a tool call that errored) is emitted here as a by-product of normal serving. Zone A also owns the graceful fallback for what the system cannot do, and the observability layer that watches traces, ratings, latency, and cost.
• Zone B, capture and bank: that signal is sanitized (deduplicated, junk filtered, and in production PII-redacted) and stored in a durable bank the axis owns.
• Zone C, offline mining: This zone comprises batch jobs that run far from live traffic and turn thousands of raw entries into a short, ranked list of work worth building, each item leaving as a spec that defines the work.
• Zone D, the foundry: a coding harness builds whatever artifact the spec ordered, with human domain knowledge guiding the harness.
• Zone E, certification and release: a set of gates fitted to the artifact, the one mandatory human approval, a staged rollout, and the replay that verifies the banked signal is now handled and closes it.

The zones stay fixed; what changes per axis is what flows through them. Coverage banks unanswered demand and builds new agents. Structure banks misroutes and overlap and builds reorganizations. Tool foundry banks tool failures and workarounds and builds repaired hands. Any failure signal worth banking can seed a fourth. Because the zones are fixed, Part 2 can reuse this map wholesale, and every axis reads as a variation on a shape you already know.

The governance rail. Running under all five zones: sandboxing and least-privilege permissions for anything the system builds, budget caps on the improvement pipeline itself, immutable audit logs, and one rule that matters more than the rest. I call it bounded recursion: the harness may never modify the certification suite, the approval gate, or its own guardrails. The system improves what it serves. It does not get to improve its own report card.

That’s the map. Now let’s walk it once, properly, on the axis I would build first.

Axis 1 — Coverage: unknown queries become certified agents

I would build this axis before anything else, because it is the substrate. The bank, the foundry, the certification pipeline, the registry, and the replay tooling you build here get reused by every later axis; once coverage exists, roughly 60% of each later axis is already standing. It also owns the most legible chart in the whole program: a falling curve of “things we couldn’t do.”

The journey: from a stranded query to a certified agent

Let me walk one such set of unanswered queries through all five zones and explain along the way how each zone actually works with an experiment implementing the same design in the below sections.

A user asks your platform something it has never handled. In Zone A, the router scores the query against every deployed agent’s capability fingerprint, a compact representation of what that agent can do, built from its description and example queries. Every score comes back below the router’s confidence threshold (τ). Instead of bluffing, the system serves the graceful fallback: “I can’t do this yet,” a best-effort generic answer, and an offer to escalate. The user moves on while the query does not.

In Zone B, that query is sanitized and banked as an open entry, along with its trace and the reason it failed. Every entry starts as open. It moves to in progress if mining selects its group for a build, and to closed only when a later replay proves a deployed capability now answers it. That plain status field is what lets the system measure itself instead of re-mining the same pain forever. Over the following weeks the entry is joined by hundreds of cousins: other phrasings of the same unmet need, queries some agent looked at and declined as not its job, sessions that got an answer but earned a thumbs-down, and escalations. That last group matters. An agent that confidently answered wrong is a worse gap than one that refused, so poorly rated sessions belong in the bank too.

When the open count crosses a threshold, Zone C wakes up. The mining job groups the backlog (two proven ways to do that grouping in a moment) and then questions every group before it is allowed to cost anyone money. Is the group coherent: one real capability, or a grab-bag? Is it new, or does it sit close to an existing agent’s fingerprint, in which case this is a routing bug and a far cheaper fix? Groups that pass are ranked by an ROI gate (could be custom determined but for this blog it is - frequency X business value X feasibility). For the winners, a research pass gathers what solving the gap would take, and the zone emits a Capability Spec whose golden eval set is drawn from the banked queries themselves. The failures that defined the gap become the exam the fix must pass. There is a trap in that: if the builder gets to study the same queries that will later grade its work, certification measures memorization instead of generalization. So the group’s queries are split at spec time into a build slice the foundry may read freely and a held-out certification slice that stays sealed, owned by Zone E, until exam day. The exam still comes from real failure and the builder still has to pass it blind.

In Zone D, the build decision comes first: new agent, extension to an existing agent, missing tool, or router-only fix? Most gaps should not mint a new agent. Every additional agent makes routing a little harder, a tax the pool pays on every future query. Whatever the decision, the foundry harness does the construction: it plans, writes the agent’s code, prompts, and tool bindings, and iterates in a sandbox until the golden set passes. This is also where human knowledge enters the build. The foundry works with domain experts the way a contractor works with a client: the organization’s policies, its tool conventions, and its knowledge of how things actually get done flow into the prompts and tool bindings being written, so the capability that emerges matches how this customer operates.

Zone E is where trust is earned. The candidate runs the gauntlet: golden-set evals, red-teaming, contract checks, regression against the pool, cost and latency budgets, with a bounded repair cycle on failure. A human reviews the certification report and signs off. The new agent enters the registry with full lineage (which bank entries spawned it) and rolls out shadow-first. Then the banked queries replay against the new agent, and entries flip from open to closed. The users who asked those questions were never answered, but the ledger now records, with evidence, that today’s platform would have answered them. A fair reader will object that closing an entry with the very capability built from it is bookkeeping and says nothing about quality, and I agree, which is why the proof of quality lives somewhere else. Certification is graded on a held-out slice of the group’s queries that the builder never saw, and the headline metric is measured on new traffic the system has never banked, so the curve only bends if tomorrow’s users are better served. In other words, replay tells you whether the backlog is being worked, while the held-out exam and the decay curve tell you whether the system actually got better. Next month, the user who started this story asks again, and this time it just works.

How to group a thousand strangers: two proven routes

Zone C’s grouping step gets its own section because it is where most naive versions of this idea die. You have thousands of unlabeled, messy, real-user queries. How do you turn them into a short list of nameable demands, with evidence that each demand is real and not an artifact of the method?

Route 1, label-first: let a language model draft the taxonomy, then apply it at scale. This is the route I would take. You ask a model to do what a human analyst would: read a sample of the backlog, propose a set of candidate intent labels with descriptions, refine and merge that label set over a few passes, and then assign every banked query to a label, with a confidence score and an explicit “none of these” escape hatch. Groups are simply the label buckets, and they are interpretable by construction: each arrives named, described, and countable. This is a published, production-proven pattern. Microsoft’s TnT-LLM (KDD 2024) ran exactly this generate-a-taxonomy-then-label-at-scale pipeline over Bing Copilot conversation logs, distilling the labels into a lightweight classifier for scale, and Microsoft’s earlier generate–validate–apply framework established the same discipline for user-intent taxonomies from search and chat logs. The discipline is what matters: refinement rounds, near-duplicate label merging, per-assignment confidence, and an abstain bucket are what separate this from “I asked a chatbot to sort my tickets.”

Route 2, geometry-first: embed and cluster. The classic: embed every query as a vector, run a density clusterer such as HDBSCAN, and read the resulting clusters as demands. Its strength is that it assumes nothing (no label vocabulary, no prior) and it can surface shapes nobody thought to name. Its weaknesses are equally well known: results sway with the embedding model and distance hyperparameters, clusters come back unnamed and need a labeling pass anyway, and short queries give geometry very little to grip. A whole research line, New Intent Discovery and its recent LLM-assisted variants (ClusterLLM, IDAS, IntentGPT), exists because making this route reliable is hard.

In practice I would run one route as primary and keep the other as a cross-check, because they fail differently: label-first inherits the model’s framing, geometry-first inherits the embedding’s blind spots. When the two agree, you can build without second-guessing yourself. When they disagree, the disagreement is useful, because it almost always points to a group that is less coherent than it first looked. Either way, every group still faces the coherence and novelty checks against the existing capability catalog before any money is spent.

Neither route is immune to vocabulary. Real users name the same thing many ways. The demand behind “travel and expense report,” “T&E,” and the brand name of whichever platform your company just adopted is one demand, but nothing on the surface of those strings says so. Geometry-first grouping splits it whenever the embedding model has not seen your organization’s shorthand. Label-first does better on public aliases, because a strong model already knows which product names refer to the same thing, but it is just as blind to vocabulary that never left your company: internal codenames, team acronyms, the tool everyone still calls by its old name. And the damage shows up in the numbers. A demand split three ways is a demand undercounted three ways; each fragment can sit below the build threshold while the true total clears it easily, and the ROI gate never sees the gap it just missed.

So the design treats domain knowledge as an input, in layers. At capture, the sanitize step maps known aliases to one canonical name using a small glossary seeded from the product catalog and the agent registry, so obvious synonyms collapse before grouping starts; this is entity linking in its cheapest form. At mining, the taxonomy-drafting prompt reads the registry, the catalog, and the glossary before proposing labels. This is a practical advantage of the label-first route: you can hand context to a labeler, and you cannot hand it to a frozen embedding. Before the ROI gate, a pairwise-similarity pass over the finished groups surfaces suspected twins to the human review that already exists, and every pair a human confirms is written back into the glossary, so the pipeline makes each vocabulary mistake about once. If a duplicate still slips through and ships, the Structure axis in Part 2 mines exactly that kind of overlap. (A public dataset contains no organization-private vocabulary, so the glossary layer is described here but was not measured.)

Vocabulary splitting is one of several ways grouping can go wrong with each of the problem having its own fix. Its mirror image is polysemy, where a single word hides two different demands: “returns” means refunds to one user and portfolio performance to another. That is what the coherence check is for, since a group is only allowed to become a build if it turns out to be one real capability. Then there is the question of how fine the labels should be, because you can go wrong in both directions, with one giant bucket that swallows everything or a separate label for every turn of phrase, and the refinement-and-merge rounds in the taxonomy step are what keep labels at the size you would actually build for. Finally, some queries ask for two things in the same sentence, which is why assignment should allow a primary and a secondary label instead of forcing a choice. None of this is unusual, and that is the point: these ordinary failures are why the grouping stage is a reviewed pipeline with gates and not a single clustering call.

Metrics

There are four numbers that tell you whether this axis is working -

Unknown-rate decay. This is the north star: the share of incoming traffic that ends up in the fallback path, tracked over weeks and months. You already have this number, because it is your no-match or fallback rate. What changes is that its decay becomes the system’s main output, and since the only clean way to bend that curve is to close real gaps, watching it fall is watching the axis do its job.

Gap-to-live time. The time from a demand first crossing the bank’s threshold to a certified capability serving traffic. This is the speed limit of the whole design, and it is where the gain over a human-only process is largest, from quarters down to weeks.

Cost per certified capability. Everything spent on mining, research, builds, repairs, and certification, divided by the capabilities that actually shipped. This is the ROI gate’s report card, and in my experience it is the first number finance asks for once you propose running the program at scale.

Out-of-scope leakage. This is the metric that catches over-reach: the share of queries that should have fallen through, because they sit outside the product’s charter, but were absorbed by an over-eager new agent.

Examples

To make this concrete, here are three scenarios of the kind this axis exists to handle, each built around a different decision the pipeline has to get right: when to mint a new capability, when to extend an existing agent instead, and how to ground a build in a system of record. All three are authored illustrations, showing the intended motion of the system. The only measured claims in this post are in the experiment section below.

The equity-program group. An employee-support platform at a growing enterprise starts seeing questions it cannot route: “when does my new grant start vesting,” “how do I enroll in the stock-purchase window,” “what happens to my units if I transfer to the Berlin office.” Individually they look like noise. In the bank, over a few weeks, they consolidate into one named group: equity and stock-program questions, a capability nobody built because the program itself is new. The novelty check confirms no deployed agent comes close to this territory; this is new demand. The ROI gate does the product math: a steady stream of queries, each currently ending as an escalation to a human benefits specialist with a real cost attached, against the one-time cost of a certified build. The foundry builds a candidate. Its golden exam is drawn from the banked queries themselves, with the held-out split described earlier, and certification adds an extra round of red-teaming because equity questions sit one wrong sentence away from financial advice. Humans appear at exactly the two points the architecture reserves for them: the benefits team answers the foundry’s domain questions while the candidate is being built (Zone D’s advisory role; the glossary they maintain is part of the same provisioning), and a human owner signs the certification report before anything reaches users (Zone E’s approval gate). After sign-off and a canary rollout, the replay pass closes the open entries, and the chart of that group’s unknown rate bending toward zero is the one I would bring to the exec review.

The expense-report migration. A finance-operations assistant lives through a company-wide migration to a new travel-and-expense platform, and “how do I file, fix, or approve an expense in the new system” queries surge. Here the interesting decision is the build decision. The new group’s profile sits close to the existing expense agent’s fingerprint: same intent territory, new system of record underneath. Minting a second, near-identical agent would make the router’s job harder on every future query. So the system chooses extend: one new tool binding against the new platform’s API and a few lines of prompt for the existing agent, certified against both the new golden set and the agent’s existing regression suite so nothing it already did gets worse. Knowing when not to create an agent is half of what this axis automates, and it is a product judgment expressed as a similarity threshold.

The retention question the BI copilot kept dodging. An analytics copilot deflects every variant of “show me D30 retention by acquisition channel,” and the bank slowly fills with rephrasings of the same unmet analytical demand. This build is different because the ground truth lives in the data warehouse. So the spec links its example queries to the warehouse schema, and the evals score execution accuracy: the generated SQL has to run and return the golden result set, not merely look plausible. For e.g - two candidates can differ by one join, one attributes every user to the channel that first brought them in, the other to the channel of their most recent visit, and both return a tidy retention-by-channel table. Only running them against the same warehouse snapshot and comparing the tables shows that one matches the data team’s reference result set and the other does not, which is why the eval grades the returned rows and never the SQL text. The ship bar at the end of the day is a product call. The data team picks an execution-accuracy threshold it is willing to stand behind, and everything below that bar gets a graceful fallback: “here’s the SQL I would have run.” Shipping at that bar today beats waiting a long time for near-perfection.

Benefits: why this beats the manual version

The obvious benefit is speed and compounding: gaps close in weeks instead of quarters, and every closed gap makes the platform a little more complete.

The deeper benefit is the one I flagged in the introduction: context. The realistic alternative to this design is a person with an export of the fallback logs and a general-purpose AI assistant. That assistant is capable, but it is outside the system. It sees the snippets someone pasted; it does not see full traces, the registry of existing agents and their fingerprints, the lineage of past builds, the golden sets that define correct behavior, or what certification rejected last month and why. Without that context its suggestions are plausible and unmoored, and each round of manual improvement drifts a little further from what the production system needs. The foundry in Zone D is wired into the bank, the registry, and the certification results, so every build decision is anchored to the system’s real state. That anchoring is the difference between self-improvement and a pile of well-intentioned patches.

There is an engineering-quality benefit too. Because every capability is written by the same harness against the same templates, registry conventions, and human-supplied domain context, good practice gets applied uniformly: prompts follow one structure, with stable system-prompt prefixes that serve-time caching can exploit; the call on what belongs in a tool and what belongs in the prompt is a codified decision; tool chains are declared and inspectable; and every artifact arrives with the same test scaffolding. Hand-built agent pools drift into a museum of personal styles. A foundry-built pool stays uniform, which keeps it cheap to review, cache, and, when the Structure axis arrives, reorganize.

Additionally, there are two more benefits that are easy to miss. The bank is a product roadmap that writes itself: a ranked, quantified, user-worded list of unmet demand that most PMs would pay dearly for. And the design leaves an audit trail by construction. Every capability traces back to the demand that justified it, the spec that defined it, and the certification that admitted it, which turns “the AI changed itself” from a compliance nightmare into a reviewable ledger.

Caveats: where this axis helps, and where it quietly stops

The gains are front-loaded. The first pass is the good part: the big obvious groups get closed and the unknown-rate curve drops steeply. Then it saturates. Once the standing backlog of buildable demand is cleared, the steady-state job is smaller: reacting quickly when new demand appears after a product launch, a policy change, a season. This is the expected outcome of the axis. After the first pass much of the ongoing value shifts to the other two axes, which is where Part 2 picks up.

Cold start is real. Grouping needs volume. Below meaningful traffic, Zone C is a human reading the bank directly, which is fine, and a good week-one ritual for any team.

Grouping quality is the critical wall. There is a published negative result worth taking seriously: an MIT/Harvard team found that a naive cluster-then-synthesize pipeline failed to beat a trivial frequency baseline for skill generation. That is why the human review of groups, the ROI gate, and the certification gauntlet carry real weight in this design, and why the grouping step gets two independent routes and a cross-check.

Not every gap is auto-buildable. Capabilities that need new third-party integrations, credentials, contract negotiations, or regulatory sign-off will exit the pipeline into a human queue at the build decision. The system’s job there is to discover and quantify the demand, which is still most of the value.

Success creates the next problem. A coverage axis that works produces agents, steadily. Left alone, the pool sprawls and the router pays for it. Which is exactly why the Structure axis exists, and why it leads Part 2.

The experiment

Everything above is a design argument. To find out whether it holds up, I built a working version of the loop and ran it against a public dataset. This section describes that run and what it showed.

The goal of the experiment was to run the whole Coverage loop live and check how much of the unknown user query pool drops. You start with a few known capabilities, and from then on, as new unknown queries start coming in, the loop fires and new agents get built that satisfy the request, post certification and human approval. Nothing is staged: the queries arrive as a stream, the system decides on its own which ones are worth building for, writes the agent, puts it through the gauntlet, and only then lets it answer real traffic. What I wanted to see was the shape of the curve. If the design works, the share of traffic nobody can answer should fall batch after batch. That curve only counts if the answers behind it are right, though, because an agent that answers confidently and wrongly would make the number fall just as well. So the run is scored after the fact by a referee that holds the dataset’s human-written labels and checks whether each query reached the right agent. The system never sees those labels during the run; it only ever sees the queries.

Let’s understand the experiment structure. The main experiment ran three times, once per random seed, and each time the system worked through ten batches of replayed traffic, building and certifying new capabilities as it went.

Here is the setup -

The experiment code can be found here - https://github.com/Yash-Dharmadhikari/rsi-operational-architecture-v2.git

The world. A world here is the whole sealed universe the run lives in: the set of things users can ask about, which of those the platform can already do on day one, which ones it is expected to discover on its own, and the stream of traffic that arrives batch after batch.

CLINC150 — a public dataset I chose for this experiment — provides 150 short, real user intents plus a curated set of truly out-of-scope queries. Each run draws 100 in-scope intents with a fixed, pre-registered seed and splits them 50/50. The 50 seed intents play the role of the platform’s existing capabilities: each gets a template-built agent, initialized from 10 labeled examples that never enter the traffic stream. The 50 discoverable intents, the ones this loop is expected to discover, get nothing: no agents, no labels, no hints. Every remaining query, with true out-of-scope traffic mixed in, is shuffled into ten replay batches of roughly 975 queries each. The whole run repeats on three seeds. By construction, roughly half the stream is unanswerable at the start.

Two details in that construction matter. The examples that build the seed agents are pulled out of the stream, so the platform is never asked a question it was shown the answer to. And the out-of-scope traffic is left in deliberately: about 2.6% of the stream is nobody’s job, which sets a floor. A run that drives the unknown rate to zero has not won; it has started answering questions it should have declined.

The stack. The experiment runs on three separate tiers.

• Serving is the tier that answers users. Every certified agent is a prompt plus a set of tools running on deepseek-v4-flash, called through a metering proxy so every token any component spends is charged to that run. This is the only tier that faces the traffic stream.
• Building is the tier that writes new agents. When mining decides a group of unanswered queries deserves a capability, the DeepSeek Harness (dsh), a coding harness pinned to an exact commit, writes the agent’s prompt, its tool code and its card, and iterates until its own build slice passes. Each build runs inside its own locked-down sandbox, so a build that misbehaves cannot reach anything outside its box.
• Routing is the tier that decides who answers what. Every query and every agent is turned into an embedding: a list of numbers that stands for meaning rather than wording(in a high-dimensional mathematical space), produced here by google/embeddinggemma-300m at a pinned revision. Two ways of asking the same thing land close together in that space even when they share no words, which is what routing needs: a user who types “my card got declined” should reach the payments agent that was described as “resolve failed transactions.” So the router scores the query against every agent’s fingerprint and picks the closest one, provided the score clears a confidence threshold. Below that threshold it refuses to guess, serves the fallback, and banks the query as unanswered. The threshold is not hand-picked; it is derived per world by a pre-registered procedure.

The gates. Left alone, a system that writes its own capabilities will happily ship confident garbage. Something has to stand between “the builder produced an agent” and “the agent is answering your users,” and it cannot be a human reading every diff, or the loop stops being a loop. The gates fill that gap mechanically: a fixed set of checks, each with a number frozen before the run, that a candidate must clear on evidence. They are also what makes the recursion safe to run at all. The builder may propose anything, but the bar it has to clear was set by someone else, in advance, and it cannot move it. Every candidate has to pass all five checks before it serves a single query in production:

• G1, the sealed exam. A held-out set of golden queries the builder never saw must route to the candidate and come back answered correctly on the serve model. At least five goldens have to reach the candidate at all, at least 80% of those must be answered correctly, and at least 95% of its tool calls must parse. This is the gate that says the capability works.
• G2, no regressions. Every capability already in the pool is re-probed with the newcomer present, and pool accuracy may drop by at most half a point.
• G3, no theft. The share of probe traffic the newcomer pulls away from capabilities that were already answering it correctly must stay under a frozen ceiling. This stops a new specialist from swallowing its neighbors. G2 asks whether the pool got worse; G3 asks whether the newcomer took work that was not the gap it was built for.
• G4, budgets. At least 90% of the goldens must finish within the tool-turn budget, and median latency must stay under the bar (about 6.5 seconds, normalized against a canary so that infrastructure hiccups do not count against the candidate).
• G5, spec conformance. The candidate’s card passes a strict schema check.

A candidate that fails gets a structured report back for a bounded number of repair attempts, then parks in the rejected gallery. Passing is still not shipping: the candidate serves one batch in shadow and is promoted only if what it claimed in shadow holds up on live traffic; otherwise it is rolled back.

The referee. Every sentence in CLINC150 comes with a topic label that a human wrote years ago, and the system never sees that column; it sees sentences only. When the run ends, a separate piece of deterministic code, the referee, uses those labels to check every served answer: was this query answered by the agent that actually owns its topic? If yes, the query counts as covered. If a different agent answered it, it counts as a miss, even though the user received a confident reply. That makes the referee stricter than the router on purpose. The router matched on meaning during the run and can be fooled by a near neighbor, so grading on meaning would repeat the mistake; the human labels are the only thing that can catch it. The referee also grades only traffic an agent never saw: an agent’s own example queries had already failed and been banked before the agent existed, so no agent marks its own homework. It holds each agent to one topic, so an agent built from a cluster that mixed two intents is charged for the one it lost, and it counts a refusal as correct only when the query was out of scope. No model, no similarity score and no API call are involved, and a test fails the build if any part of the system other than the world builder and the referee so much as mentions the label column.

Scoring coverage. An agent that answered confidently and wrongly earns nothing. Scored this way, the in-scope unknown rate fell 28.0%, 22.8%, and 25.4% relative by seed, a 25.4% mean. That closes about a quarter of the gap.

Why 25.4% is a floor. There are two deliberate choices that make 25.4% a floor; the first is a strict scoring mechanism. A query counts as covered only when the answer matches the true intent exactly, so an answer that lands on a sibling intent, which is what 65.2% of the “wrong” answers did, counts as a miss even when a product team would call it a hit. That makes 25.4% the least the system achieved. The second is strict admission. Only capabilities that cleared all five gates and a shadow batch were allowed to serve, which means the candidates the gauntlet rejected, and the capabilities the shadow trial rolled back, would have added coverage if they had been let through. That coverage was left on the table on purpose. What actually set the floor was how much the router could find, and the next paragraphs explain why.

The three seeds ended with 42, 44, and 39 capabilities live; 34, 20, and 25 in shadow; 15, 18, and 17 rejected by the gauntlet; and 9, 5, and 9 rolled back after their shadow trial. Closure rate is scored the same way, with a banked entry counting as closed only when replay routes it to the capability that owns its intent, and it came out at 27.3%, 14.8%, and 22.1% per seed.

What the gates did in this run. The gauntlet was on for the whole main run, and it was busy: the rejections above happened outright, and the rollbacks came from candidates that had passed the gates but did not hold up on live traffic. The more useful question is where the rejections came from. Nearly every candidate the gauntlet turned away had answered every golden query that actually reached it, and 45 builds across the three seeds died this way. These were failures of delivery, since the router never sent the exam questions to the candidate, so G1 failed it for being unfindable. In this run those builds were still built, and paid for, before G1 turned them away; and I added the reach check that refuses them up front afterward. The gate was doing its job, because a capability nobody can reach is worthless in production.

The router owns part of this failure. It would be convenient to read the whole shortfall as a coverage problem, and it isn’t one. A capability that was built correctly and then never reached is a retrieval failure, even though the books record it as a construction failure. The gauntlet turned candidates away for being unfindable, so the miss is shared: the build tier did its job, and the part of the system that decides who answers what did not get the new capability in front of the queries it was built for. No amount of better building fixes that half.

So the improvements worth considering next are routing improvements. Fingerprints could be built from the banked queries a capability was created for, in the users’ own words, instead of from the spec’s tidy examples. The confidence bar could be set per capability instead of once for the whole pool, since a narrow specialist and a broad generalist do not deserve the same threshold. Routing could happen in two passes, a wide shortlist first and then a closer look at the handful of candidates, so a near miss gets a second opinion instead of a silent decline. Sibling intents that keep stealing each other’s traffic could be separated deliberately, by teaching the router what makes them different. New capabilities could be indexed the moment they ship, so nothing waits a batch to become reachable. And the reach check, which today refuses to pay for builds it can prove unroutable, could move from a guard at the end to an input at the start: if the router cannot reach a group, that is a routing job before it is a building job. Part 2’s Structure axis is where that work gets its own loop, its own signal and its own gates.

The system did what it was built to do. Over ten batches, the share of questions nobody could answer fell from 56% to 42%, with every new agent checked for correctness before it was allowed to serve, and most of what stayed unanswered was not for want of building. The agents were there; the router could not find them. That is where the next part of this series starts.

What comes next: the other two axes

Coverage comes first, and in turn it creates the problems the other two axes exist to solve, each of which improves the system along its own dimension as well. They do not wait in line; once the substrate is standing, they run in parallel. Structure is the counter-force to growth: it watches the pool coverage keeps enlarging, mines the evidence of disorganization (overlapping agents, misroutes, dead weight), and executes certified reorganizations with a zero-coverage-loss proof, so routing stays sharp as the system scales. Tool Foundry is the multiplier: it reads tool errors, retry storms, and agent workarounds as a friction signal, then repairs or builds the tools every agent shares, so one fix upgrades the whole pool at once.

Both run the same five zones you have just watched coverage run, with different signals in the bank and heavier gates in certification.

What’s deliberately not novel here

No component in this post is new. The claim is about composition: every stage exists somewhere, in production or in the literature, and what I could not find, as of early September 2026, is all of them wired into one certified, self-closing system. Here are additional details of the previous work in this area; the list is not exhaustive:

The closest commercial systems now build a fix. LangSmith Engine clusters production traces into issues, drafts a prompt or code change as a pull request, and proposes the evals that would catch a regression. Decagon’s Duet Autopilot mines conversations and escalations, drafts procedure changes, and tests them in simulation against a golden set before a person approves each one; Decagon’s own DuetBench scores it on refining existing agents. Both stop at editing an agent that already exists, and both leave the deploy to a human. The vendors’ most recent moves are on the gates. Sierra’s release governance brings checks, approvals and staged rollouts to the changes human builders make, and IBM’s watsonx Orchestrate self-improving agents regenerate an agent’s instructions, re-test them against your test cases, and let you push the result. Each hardens one or two stages of the skeleton. None of them to my understanding notices a capability the platform lacks and builds it.

The research on self-evolving agents supplies the argument for certification-first. EVOMAL shows that coding agents which learn by imitating skills from a shared library will copy a planted malicious skill into new skills of their own, at rates between 20% and 42% across six models, and that the copies persist after the planted skill is removed. A library that admits self-authored skills without a gate is a propagation path, which is what G1 through G5 and bounded recursion exist to close. The constructive half is there too. WikiSkill co-evolves skills with a persistent knowledge base built from agent experience, and a study of feedback dynamics in evolving skills finds that every evolved skill worth keeping came from feedback that included failed trajectories, with a validation step deciding which candidates survive; that is the bank and the gate, on benchmarks. A survey of self-evolving coding agents and a paper asking whether prescribed optimization pipelines are still needed frame the open question this series takes a position on: how much of the improvement loop should be fixed machinery, and how much should be left to the model. My answer is the governance rail. The model may propose anything; the gates, the approval and the rollout stay fixed.

The existence proofs that the pattern works at production scale are: a July 2026 report of a certified efficiency cycle running eight months in production (Progressive Crystallization: 45% of executions crystallized into deterministic workflows, per-incident agent cost down more than 70%), and NVIDIA’s EACL 2026 account of a MAPE-control-loop data flywheel on an internal assistant serving 30,000+ employees (routing model swapped from a 70B to a fine-tuned 8B at 96% accuracy). Cursor’s Tab model ships an online preference-learning cycle. Each is one axis, and none generalizes the family.

And the recurring question, answered once: “isn’t this just RL?” It has the shape of RL. The bank plays the role of prioritized experience replay; certification is safe policy improvement expressed as constraints; staged rollout is a trust region. But the improvement operator is program synthesis by a coding harness, there is no scalar reward, and nothing explores on live users. The nearest ancestor is a MAPE control loop, not Preference Policy Optimization or DPO/GRPO and the likes of it.

Summary

What Coverage taught me about RSI as an operational architecture. The useful thing the Coverage axis showed is not that a system can grow its own agents, it is that self-improvement only works as an operational architecture: a signal you can trust, a builder that produces candidates, and a certification pipeline that decides which candidates are allowed to serve, all wired into the same five zones and governed by the same gates. Take any one of those away and you have either an experiment or a liability. Keep all three and the system becomes something a team can run, audit and hand over, the same way they run a deployment pipeline today.

The first measurement costs you nothing, because your unknown rate is already sitting in your analytics as a no-match or fallback percentage. You have had the number all along; what changes is treating it as a backlog.

The thesis underneath all of it is that the skeleton is the product. The five zones together, from the live serving in Zone A that catches what the system cannot answer, through the bank, the mining and the foundry that turn those misses into candidate agents, to the certification and release in Zone E that decides which of them may serve and confirms the miss is closed, are what turn “AI that changes itself” from a liability into a system you can audit, and they are reused unchanged by every axis. Coverage is the first thing that runs through them, and each axis after it is a configuration of the same skeleton with its own signal, its own builder and its own gates.

Part 2 walks Structure and Tool Foundry through the same skeleton. Beyond those, there are four more axes in my current research (telemetry, federation, anticipation, autonomy calibration), each with its own signal, gate set, and KPI. I am planning to add them as later parts, if the series earns it well. Thank you for reading !

Citations & further reading

Surveys and framing. Self-Evolving Agents survey · Self-Improvements in Modern Agentic Systems · Recursive Self-Improvement in AI (July 2026 survey) · Self-Evolving Coding Agents (survey, Aug 2026) · Rethinking Self-Evolving Agents: Do We Still Need Prescribed Optimization Pipelines? · NVIDIA — Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement (EACL 2026 Industry) · Progressive Crystallization · Pioneer Agent

Grouping and intent discovery. TnT-LLM: Text Mining at Scale with Large Language Models (KDD 2024) · Using LLMs to Generate, Validate, and Apply User Intent Taxonomies · CLINC150 / out-of-scope intent dataset · DeepAligned — New Intent Discovery protocol · ClusterLLM · IDAS · IntentGPT · Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining (negative result)

Meta-agents and self-improving systems. ADAS / Meta Agent Search · Darwin Gödel Machine · Alita · Alita-G · EVOMAL: Self-Poisoning in Self-Evolving Coding Agents · WikiSkill · Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds

Industry systems referenced. Sierra — Agent Development Life Cycle · Sierra — Release governance · Decagon Duet · Decagon Duet Autopilot · DuetBench · IBM watsonx Orchestrate — self-improving agents · Kore.ai Intent Discovery · Microsoft Copilot Studio Themes analytics · IBM’s deployed unrecognized-utterance pipeline · LangSmith Insights Agent · LangSmith Engine · Braintrust Loop · Dialogflow CX no-match handling · opencode · CLINC150 on Hugging Face