Productizing Loop Engineering

Loop Engineering is a way of accomplishing tasks in which you give an agentic harness an objective function (or a reliable proxy for one) and a set of tools, and let it iterate: produce an output, measure it against the objective, feed the measurement back, and produce a better output, repeating until the objective is satisfied. The human’s job shifts from doing the work to defining what “good” means; the loop does the producing, the measuring, and the improving.

That definition sounds simple, but it hides a large design space: what exactly the loop optimizes, how honestly it is scored, and when it is allowed to stop. Those three choices are what separate a demo from something an enterprise can rely on, and they are the subject of this post.

The current state of Loop Engineering

Most of what we currently call a “loop” is an execution loop: an agent takes an action, observes the result, adjusts, and repeats until the task is complete. This is the pattern behind ReAct, function-calling agents, and tools such as Claude Code’s /loop, and it has become genuinely effective at carrying tasks to completion. Alongside it sits a second, longer-established family, the optimization loop, which underpins AutoML, neural architecture search, and modern prompt optimizers, and whose goal is not to complete a task but to search for the artifact that best satisfies a metric. Much of the recent progress in Loop Engineering comes from bringing these two together: using an autonomous agent not merely to finish a task, but to synthesize and select a solution against an objective. The sharpest demonstration of that convergence — and one that has been hugely inspirational for my own thinking here — is Andrej Karpathy’s autoresearch: an agent handed a small training script, a single objective metric, and a fixed time budget per experiment, left to iterate overnight — and it found genuine improvements in a model he had spent years tuning by hand. Each family, however, was designed under an assumption that becomes limiting once they combine. Execution loops typically stop when the result appears satisfactory, and optimization loops typically select a winner by scoring against the same data used to guide the search. As a result, these systems are often good at producing a solution but less equipped to tell us how well it will hold up outside the loop, or what it will cost to run.

These two gaps, generalization and economics, are where the practical challenges concentrate. When a loop is evaluated on the data it was optimized against, it tends to fit that specific sample, so a passing score can reflect the evaluation setup as much as the underlying quality; separating the two requires evidence the loop never had access to. Cost raises a parallel issue: many systems optimize for accuracy alone, while in deployment the real objective is usually accuracy subject to a budget for compute, latency, and human involvement; whether each individual decision the system makes — one ticket triaged, one invoice matched — costs $0.25 in compute or $5.00 in a person’s time often determines whether an automation is viable at all.

Addressing both gaps starts with a simple separation of roles: one system searches for solutions, and a different system grades them — an examiner the searcher cannot see into, holding data the searcher has never touched. With that split in place, a few concrete requirements follow naturally. Scores stay independent of the search, because the judge is sealed. Cost becomes an explicit objective the search optimizes, not an afterthought to accuracy. And no winner is trusted until it earns a held-out certificate: proof that it performs on data the loop never saw, rather than just on its own practice set. But the requirements are only the guardrails. The real prize is what they make safe to attempt: a loop free to generate novel agent architectures in a complex action space — not just tune the knobs on a design a human already chose — while hard accuracy and cost constraints keep every invention honest, and a held-out certificate decides what survives. A search loop built that way is what the rest of this post develops, and what I demonstrate end-to-end on a concrete IT Service Management (ITSM) triage task.

Why Loop Engineering? The advantages

When the conditions are right (and I’ll be specific later about what “right” means), a well-run loop has advantages that are hard to match with hand-built automation:

• It searches a design space no team would staff. Even a modest automation hides dozens of coupled design decisions: model choice, prompt wording, confidence thresholds, ensembling, escalation policy, retry logic. Those multiply into thousands of viable configurations. A human team hand-tunes a couple of them and ships. A loop can evaluate dozens of candidates systematically, overnight, and keep the receipts for every one it rejected.
• It needs minimal human involvement while it runs. People define the goal, the constraints, and the data once, then review a finished, evidence-backed result. Senior engineers spend their time on judgment (“is this the right quality bar?”) instead of on iteration (“did threshold 0.48 beat 0.50?”), which is where their time actually compounds.
• The harnesses are already good enough. Agentic harnesses like Claude Code and Cursor already perform remarkably well when given a defined objective, the relevant tribal knowledge, and a measurable goal to optimize against. “Loop until the metric passes” is a natural generalization of the “loop until the tests pass” workflow they execute every day.
• It sweats details engineers routinely miss — and the details are where the money is. The Manus team’s write-up on context engineering shows how much production-agent economics hinge on decisions few designers think about: with Claude Sonnet, a cached input token costs roughly $0.30 per million versus $3.00 uncached (a 10× difference), and a single stray token, like a timestamp in the prompt prefix, can invalidate the KV-cache for everything after it. Manus rebuilt their agent framework four times as they internalized lessons like these. A loop that meters real cost and latency doesn’t need to be told any of this: a design that breaks the cache or bloats the context simply scores worse, and the search walks away from it automatically.

The proposition: Loop Engineering in production

Here is what I am proposing enterprises build. It is two components and a contract between them.

The first component is an evaluation and governance framework: a system that can take any candidate automation and grade it against the enterprise’s own data, its own cost realities, and its own constraints, and that also owns everything that happens after the grade: certifying the winner, deploying it, safeguarding it, and watching it in production. The second is an agentic harness that generates those candidates. The harness produces a candidate; the framework grades it; the grade, and nothing else, flows back to the harness; the harness produces a better candidate. The process repeats until the goal is met.

Because I’ll refer to these two constantly, let me give them shorter names: the Builder, the agentic harness that generates candidate agents, and the Control Plane, the framework that grades those candidates and then certifies, deploys, and safeguards whatever wins. The name is borrowed from distributed systems, and it earns the analogy: like Kubernetes’ control plane, it is the layer that admits workloads, enforces policy on them, schedules them into production, and watches them for as long as they run.

What does “the goal” look like? In its most general form: given a dataset that encodes what correct looks like, produce the best possible agent that passes it. And “agent” here does not mean a prompt with a model behind it; these are genuine architecture searches:

• A security-operations agent: given years of analyst-triaged alerts and an adjudicated test set, produce an agent that works each incoming alert, pulls the right context (retrieval over threat intelligence and past incidents, structured lookups against the asset inventory and identity directory), closes the false-positive tail automatically, and escalates genuine threats. The search space is real architecture: which enrichment tools to wire in, how much context to pack into each step, a single agent versus a planner with executors, and where the analyst hand-off sits.
• A code-review agent: given a repository’s history of merged pull requests and the review comments senior engineers left on them, produce an agent that reviews new PRs the way the team actually reviews them. Getting there means choosing how much of the codebase to retrieve per file, which static-analysis tools to wire in, a single-pass reviewer versus a planner-and-critics architecture, and a calibrated sense of when to stay silent. Teams hand-tune those decisions by feel; a loop can search them against the review history instead.
• An analytics agent: given analyst-approved question-and-answer pairs over the warehouse, produce an agent that plans multi-step SQL, verifies its own results, and knows when a question is out of scope. Schema retrieval, text-to-SQL tool configuration, self-verification loops, sub-agents for visualization. These are exactly the deep, multi-node architectures frameworks like LangGraph’s Deep Agents exist to express, and exactly the design space a Builder should explore against a score rather than a hunch.

In production, though, the goal can never simply mean “get the highest possible accuracy.” The Control Plane must push the Builder toward agents that also satisfy the constraints that decide whether software actually ships:

• a quality SLA: accuracy overall, and precision on the classes where mistakes are expensive;
• cost: training cost, inference cost per decision, deployment cost;
• latency: a p95 target, not just an average;
• infrastructure fit: memory footprint, hardware, and the stack production actually runs;
• an escalation budget: how often the agent is allowed to fall back to a human;
• compliance: data residency, auditability, PII handling.

The Control Plane grades every candidate against these constraints and returns a scorecard. The Builder takes the scorecard and continues what is, at heart, a well-established constrained-optimization exercise: producing better and better agents until one meets the goal while satisfying every constraint.

The economics of the design. Notice what this asks you to pay for: you deliberately spend money up front (frontier-model time, GPU hours, thousands of evaluation runs) to produce an artifact that is cheap forever after. The search is a one-time capital cost; the winner is an operating asset, and the upfront spend is amortized over every decision it makes in production. That trade is exactly worth it when the output is reusable for a long time, a real applicability condition I’ll return to near the end.

The Control Plane: evaluation, certification, and governance

To productize Loop Engineering, a company builds the Control Plane once, catered to its own constraints and objectives, and lets the Builder operate inside that defined action space. Here is what a production Control Plane contains. The list is not exhaustive, but nothing on it is optional.

The tools it exposes

• Metered actions. A selection of LLMs (and their key parameters) where every call is tracked for cost, latency, and tokens. Critically, the Builder can see the price list, so cost becomes something it optimizes rather than something it discovers after the fact.
• Prebuilt pipelines. RAG, text-to-SQL and structured search, document extraction and OCR, embedding services, classifier-training utilities: pipelines the platform team has already built, with their configuration exposed for the Builder to consume on the data it is given. Exposing a RAG tool, for example, means exposing the parameters that control which columns of the dataset get indexed.
• A human-escalation channel with a real price on it. Handing a decision to a person is a legitimate tool (often the winning move for the hardest cases), but it must carry its true cost, so the Builder economizes on it instead of treating human time and attention as a free resource.
• Sandboxed compute. Code execution, a standard ML stack, and a training budget, so the Builder can build models of its own rather than only orchestrating APIs.

Data custody

The Control Plane, not the Builder, owns the data, and it splits the data by role: a labeled build set the Builder can see and learn from; a hidden search split, which produces the iteration scorecards; a hidden calibration split, reserved for the statistical certificate; and an untouched test split, which grades the final winner exactly once. Separating the data you learn from, from the data that grades you, is the entire defense against a loop quietly overfitting its own exam.

Scoring and certification

• Score deterministically. The same candidate must always receive the same score, or the search ends up chasing noise instead of signal.
• Assume the Builder will reward-hack, and design so it can’t. Costs come from the Control Plane’s own ledger, never from the candidate’s self-report; evaluation data stays hidden; and anything that looks like fitting the exam’s quirks gets caught by the held-out splits.
• Verify constraints with real meters, not an LLM’s opinion. Dollars from the ledger, p95 latency from wall clocks, memory from the running process. An LLM-as-judge belongs only where the metric is inherently qualitative, and even then, calibrated against human ratings.
• Demand a statistical certificate. Passing once is not enough; the winner must pass by a margin large enough to rule out luck: for example, a 95% lower confidence bound that still clears the bar.
• Apply a winner’s-curse guard. The cheapest candidate that passes the search split is precisely the candidate most likely to have gotten lucky on it. The selection rule must require certifying on the calibration split and confirming on the untouched test split before crowning anything.
• Enforce hard budget caps on the search itself, so an enthusiastic Builder cannot spend its way past the experiment’s own economics.

Deployment

Once a certified winner exists, the Control Plane should hand over a deployable artifact, not just a demo: packaged for the same stack production runs (container, model-registry entry, API spec), with monitoring hooks for per-decision cost, latency, and confidence, plus a defined rollback and re-certification path. If the winning agent has to be rebuilt by hand to fit production, the loop’s advantage is drastically reduced, so production compatibility should be one of the constraints the Control Plane enforces during the search, not an afterthought.

Safety and compliance

• Prompt-injection resistance. Deployed candidates will process untrusted text (tickets, emails, invoices) that can contain adversarial instructions. The Control Plane’s evaluation sets should include injection attempts, and a candidate that follows instructions embedded in data should fail.
• Data governance. PII masking and data-residency policy enforced inside every metered tool, so the Builder physically cannot route data somewhere it shouldn’t go.
• Auditability. Every decision the deployed agent makes is logged with the model version, confidence, and cost behind it; every certificate the Control Plane issues is reproducible after the fact.
• Least privilege. Candidates and the Builder run sandboxed during the search, with exactly the access the task needs.
• Version pinning. The certified artifact is frozen (weights hash, dependency lockfile), and any change to it triggers re-certification.
• Human escalation as a safety valve. Budgeted and monitored, never eliminated. Some inputs will always sit outside what the agent can safely decide — genuinely novel situations, ambiguous edge cases, calls with consequences the automation shouldn’t own — and routing those to a person is the system correctly recognizing its own limits, not failing. The goal is to keep that route priced, observable, and deliberately used: the escape hatch is part of the design, not a failure of it.

Analytics

Per-decision cost, latency, and confidence dashboards tracked against the constraint budgets they were certified under; escalation-rate tracking; input-drift monitors wired to re-certification triggers; cumulative savings against the incumbent baseline; and search telemetry: candidates tried, prune reasons, and the cost/quality frontier over time. That last one is management’s window into what the loop did and why the winner won.

The sealing principle

Above everything else, the Control Plane’s internals must be closed to the Builder: not the hidden labels, not the scorer’s code, not the provenance of the thresholds. The only thing that crosses the boundary is the scorecard. This is what makes the final number believable: a result produced by a builder that could not see the exam is evidence; a result produced by one that could is marketing.

The Builder: the agentic harness

The Builder is the part that actually generates outputs (AI agents, classifiers, whole pipelines), operating strictly within the bounds the Control Plane defines. To do that well it needs the right tools, the tribal knowledge, and all the necessary context: the domain documentation, the naming conventions, the infrastructure realities, and a memory of what it has already tried and why those candidates were pruned, so the search moves forward instead of in circles.

This is where harnesses like Claude Code, Cursor, and LangGraph’s Deep Agents fit. They already ship the pieces this role demands: file and code tools, sandboxed execution, self-debugging, long-horizon planning, sub-agents, and hard-won context-engineering discipline. That is what it takes to reliably generate a working piece of software against a specification. Loop Engineering doesn’t ask them to be anything new; it asks them to point those capabilities at a scorecard.

One recommendation I’d make without hedging: give the Builder the most powerful base model available. The quality of every iteration compounds across the search, and a stronger model converges in fewer of the expensive evaluation cycles. Economizing on the Builder’s brain is a false economy: the whole design already concentrates spend in the search phase precisely so production can be cheap.

Proof: running the loop end-to-end on ITSM triage

To test this design I ran the full loop on a concrete problem: IT service-management ticket triage. Every help desk drowns in tickets — “the VPN is down,” “please give me access,” “we keep hitting the same crash” — and before anyone can act, each ticket must be sorted into the right ITIL category: Incident, Request, Problem, or Change. A trained person sorts very accurately but costs about $5 of time per ticket; software is nearly free but makes mistakes.

What is the cheapest way to sort tickets that still meets the quality bar — and how do we prove it before trusting it?

The setup followed the blueprint exactly. A sealed Control Plane held the tickets with hidden answer keys, a price list mirroring reality (in-house models ≈ free, commercial LLM calls priced per token, humans at $5), and the rulebook: at least 83% accuracy, at most $0.60 per ticket on average, fast answers, and a memory budget the target hardware can hold, plus a 95%-confidence statistical certificate on a split the search never touched. The Builder (an agentic harness with a GPU-NVIDIA DGX Spark, 6,000 labeled example tickets, and no access to the Control Plane’s internals) searched for the cheapest certifiable agent. The whole experiment is on GitHub — Yash-Dharmadhikari/loop-engineering-poc — if you want to walk through the search, the scorecards, and the certificates yourself.

What the search discovered, in order:

• “Just ask a frontier LLM” cannot pass. Zero-shot, a top-tier model reaches 71–75% (well under the 83% bar), because it doesn’t know this company’s labeling conventions. No amount of money fixes that; the naive approach fails at any price. Prompt engineering has the same ceiling: a few-shot prompt can show the model a dozen curated examples, but it cannot internalize labeling quirks spread across thousands of tickets — those conventions live in the data’s long tail, beyond what any context window’s worth of exemplars can teach.
• “Escalate everything to humans” passes and bankrupts you. 100% accurate, $5 per ticket, roughly eight times over budget. Together with the first result, this reframes the problem: accuracy was never the hard part. Passing cheaply is.
• One in-house model isn’t enough either. A single classical classifier — TF-IDF features feeding a linear model — trained on the 6,000 labeled examples reaches about 79% at essentially zero cost, better than the frontier LLM, still under the bar. The labels themselves are noisy enough to cap any single model near 80%.
• The winning shape is a committee plus a triage nurse. Several in-house models vote, each with a confidence level. When the committee is confident (the overwhelming majority of tickets), commit the answer for fractions of a cent. When it hesitates, don’t guess: hand the ticket to a human. The skill is entirely in choosing which tickets to hand over. A classical two-model version of this — a pair of TF-IDF-family text classifiers, cheap enough to run on CPU — passed, certified, and at production scale delivers the bar at about $0.44 per ticket: the champion to beat.

Then came this run’s contribution: the Builder LoRA-fine-tuned a mid-sized multilingual transformer on those same 6,000 tickets and added it to the committee as its most-trusted voter. The surprise: the fine-tuned model is not more accurate than the older models (most of its mistakes fall on the same genuinely ambiguous tickets where even the answer key wavers), and on its own it is actually a worse deal, needing 10.8% human escalations at $0.54 per ticket, costlier than the champion it was trying to beat. But it reads tickets differently, so it makes different mistakes. Ticket-level comparison shows roughly 80% of the two champions’ machine errors fall on the very same tickets, and all the economic action lives in the remaining margin. In a committee, a differently-wrong voter does two things there: it breaks ties (tickets the classical pair split on, which used to be automatic $5 escalations, now clear the commit threshold), and it raises red flags (tickets the classical pair would have confidently committed in error get dragged below the threshold, where a human catches them).

The Control Plane certified the LoRA committee with a stronger statistical margin than the classical champion (95% lower bound 79.5% vs 78.6%), and the production-scale check (re-scoring every candidate on the ~13,400 labeled tickets the experiment had never touched) is where the economics become trustworthy. That split deserves a pause, because it deliberately inverts ML convention. A standard experiment hands most of the data to training and holds back a sliver for testing; here the Builder saw only 6,000 tickets while more than twice that many stayed hidden. The inversion is the point: in production, the volume an agent faces always dwarfs whatever it was built on, so the honest test of the economics has exactly this shape — build on a little, then prove it on a lot. Both champions hold the bar there (classical 84.3% accuracy, LoRA committee 83.9%), while the LoRA committee escalates about a sixth less often (7.3% of tickets vs 8.8%) and runs 16% cheaper per ticket. Underneath, the swap is visible at ticket level: the committee stopped escalating tickets the classical pair had merely been unsure about, newly escalated a few it now distrusted (catching errors that used to be committed silently), and flipped a handful of votes, accuracy gains and losses cancelling almost exactly while the escalation rate fell. A buyer who values accuracy over cost can instead take the committee at a stricter threshold: 85.2% accuracy at $0.51 per ticket, a certified alternative on the same frontier. And one design detail keeps the whole trade safe: with vote weights of 1.0 (the LoRA model), 0.7 and 0.5 (the two classical TF-IDF-family voters), the commit threshold of 0.48 sits just above the largest vote share LoRA can muster alone (≈ 0.455): no ticket is ever committed on the new model’s say-so without corroboration from a classical member.

It is worth stating plainly: this run is a demonstration of the Builder–Control Plane contract, not an exhaustive search. The loop explored what its budget allowed; it did not, for instance, systematically tune prompts for the frontier model or exhaust every ensembling scheme, and a larger search budget could well push the frontier further. What the run shows is the mechanism working end to end: sealed scoring, honest economics, and a certificate you can take to production.

The scale check also vindicates the pruning. The looser threshold that undercut the winner on price during the search fails the quality bar outright at scale (82.0% accuracy: the winner’s curse, exactly as the guard predicted from a fraction of the data), and the most accurate setting breaks the $0.60 ceiling there (13.1% of tickets escalated). The candidates the loop refused to crown really were traps. That refusal is the selection rule doing its job: no candidate is crowned unless it passes the search split, certifies on the calibration split, and confirms on data it has never seen, and every pruned candidate carries its reason in the logs.

The punchline for this post is not the 16% per-ticket cost reduction. It is the shape of the winner: a fine-tuned model that loses on its own but wins inside an ensemble is exactly the kind of candidate a human team rarely bets on up front — and a sealed, certifying loop found it, priced it, and proved it without anyone steering.

Where Loop Engineering works — and where it doesn’t

Loop Engineering can produce remarkably optimal outputs, but it is not the right tool in every setting. The conditions that make it work:

• The amortization must be real. The Builder’s upfront spend only pays off if the winning agent is reused long enough for its per-decision savings to repay the search. A workflow that will be redesigned next month doesn’t qualify; a triage pipeline that runs on every ticket for a quarter does.
• The data must hold still. In settings with constant data drift (fraud patterns, trending content, fast-moving product catalogs), the loop’s winner decays, because it was certified against a distribution that no longer exists. Drift monitors and scheduled re-runs can manage slow drift; fast drift breaks the economics.
• You need objective data, or an honest proxy. The loop optimizes whatever it is scored on, so the score must genuinely encode what you want. A human-curated acceptance-test dataset, like the ITSM one above, is a good proxy: it directly captures what the business expects, and moving the metric is impossible without doing the actual job. Bad proxies fail exactly that test: optimize marketing copy against open rates and you breed clickbait subject lines; optimize support automation against time-to-close and tickets get closed prematurely; optimize a coding agent against “tests pass” on a thin test suite and it learns to special-case the tests. If gaming the metric is cheaper than doing the job, the loop will find the gaming — it is, after all, an optimizer.
• Labeling must be consistent. When enterprises don’t have objective data and buy it via annotation, label quality becomes the ceiling on everything downstream: in my experiment, inconsistent labels capped every single model near 80% no matter the architecture. Invest in labeling QA before investing in the loop; no search can out-optimize a noisy answer key.
• The search space must be constrained. The loop works best when the Builder operates like a distilled, experienced engineer inside production’s real boundaries, not an unbounded explorer. Left unconstrained, a Builder will happily produce a winner that needs a GPU cluster and three different model providers when production is a CPU-only environment with one approved vendor: optimal in the lab, undeployable in reality. This is not a reason to avoid the loop; it is the reason the Control Plane’s constraint set exists.

Use cases and domains

I think the real question is where to run the loop. The wins come wherever the two ingredients already exist — an adjudicated history that defines what correct looks like, and a real price on every decision. Here is where I would point a Builder first:

• Inference serving and cost engineering. A serving stack is a design space wearing a config file: quantization level, KV-cache and prefix-caching policy, speculative-decoding draft models, batching windows, and cascade routing that sends easy requests to a small model and reserves the frontier model for the hard ones. Set a quality floor, meter p95 latency and cost per thousand requests, and the loop tunes the stack against your real traffic instead of somebody else’s benchmark.
• Retrieval and RAG pipelines. Chunk size and overlap, embedding model, lexical-versus-vector weighting, reranking depth, context-packing order — coupled decisions most teams settle once by folklore and never revisit. A golden set of question-answer pairs makes them searchable: answer quality is the bar; token spend and retrieval latency are the constraints.
• Training recipes and fine-tunes. The autoresearch pattern pointed at your own models: LoRA rank and placement, learning-rate schedules, data mixtures, sequence lengths — each candidate a time-boxed training run scored on held-out validation data, with the GPU-hour budget as the hard cap the search cannot spend past.
• Document-driven business workflows. Invoice extraction and matching, claims routing, support-response drafting: anywhere an adjudicated history encodes what correct looks like and every decision carries a price, the same contract applies — the Control Plane holds the history and the budgets, and the loop searches for the cheapest agent that certifies against them.

Closing thought

The industry has spent the past year proving that agents can do tasks. The next phase is harnesses that build the systems that do the tasks — and the difference between that being a demo or a discipline is everything this post put inside the Control Plane: real meters, sealed data, statistical certificates. Spend intelligence up front; deploy certainty. The Builder is the engineer. The Control Plane is the engineering.