We Were Already Doing Loop Engineering. We Just Didn't Know It.

"If you are still prompting, we are dead."

I didn't know what he meant. I'd already integrated Karpathy's autoresearch concept into the system — autonomous iteration with adversarial review and convergence criteria. But "loop engineering" was a different frame. A colleague said it like it was obvious — like I should have already moved past something fundamental. He said to research it. So I did.

What I found was both validating and humbling. Validating because the governance system I'd spent hundreds of hours building already embodied most of what the research describes. Humbling because the gaps I'd been ignoring — the ones that kept biting me in demos and making me ask "why can't you get this right after three tries?" — were exactly the problems loop engineering solves.

This is the story of how we got here, what we learned, and where it goes next.

The Evolution: How We Got Here

When ChatGPT launched in late 2022, I started using AI for coding the way most people did — as a better Stack Overflow. "How do I parse a C-CDA XML document?" Immediate answer, no context switching, faster than searching forums. But the code it produced wasn't worth the time to debug. You'd spend as long fixing hallucinated APIs and subtle logic errors as you would have spent writing it yourself.

That changed when Claude arrived. For the first time, AI-generated code was consistently correct enough to use in production — not just syntactically valid, but architecturally sound. The quality gap between "interesting demo" and "real engineering tool" closed practically overnight.

Then MCP connections changed everything again. Suddenly the AI could pull database state, server logs, and configuration files together in one context. We started solving production issues that humans couldn't correlate manually — the kind where the answer lives across 4 systems and 6 log files.

That earned enough trust for the next step: building components. Well-defined modules, one at a time, with human review after each. An ETL connector. A measure engine. A frontend page.

Then whole systems. Not one component — the full stack. Architecture, data model, API, frontend, deployment. Simulating a traditional development team at 10x speed. Architect drafts the design. Developer implements. Tester verifies. Security reviews. The AI played all the roles.

But speed without governance produced garbage that looked like gold until you inspected it closely. The code compiled. The build passed. But the quality score showed a fraction of the expected value. A materialized view was stale. The API reported zero where the pipeline had computed a nonzero result. The AI said "done" when the system was internally inconsistent.

That's when we built the governance layer — stage-gates, adversarial review, domain expert personas, persistent session state. Not to slow things down, but to catch the 40% of errors that "it builds" doesn't catch.

And now we're adding the final piece: automated verification loops that run without asking permission. Not replacing the governance — augmenting it with mechanical consistency checks that catch drift between pipeline runs.

Each stage earned the right to exist by solving the pain of the previous one.

What Loop Engineering Actually Is

Three components. Miss one and you've built a money burner:

Verifier — an automated gate that isn't the AI grading its own homework. Tests, metrics, schema checks. Something that says "no" without human intervention.

State — persistent record of what's been tried and what failed. Survives session boundaries. Tomorrow resumes instead of restarts.

Stop condition — "done" must be definable. Verifier passes, or you hit a retry limit. Open-ended goals without measurable completion are just prompting wearing a loop costume.

In March 2026, Andrej Karpathy demonstrated this with autoresearch — 630 lines of code that ran 700 ML training experiments in two days, finding 20 improvements humans missed. The key insight: with an objective metric, you shouldn't be the experiment runner. Remove yourself from the inner loop and let verification drive search.

The Real Pain Point

Let me be direct about where this comes from.

I spent hundreds of hours building the governance harness — personas, gates, steering files, session state — not because I love process, but because I was tired of correcting the AI on the same mistakes repeatedly. The same stale data left behind. The same hardcoded values that should be database-driven. The same "it builds" confidence when the output was clinically wrong.

The harness simulates all the checks and balances of a real development team: the tech lead who catches architecture drift, the QA engineer who tests the output, the domain expert who says "that's not how CMS works." It reduced the frustration significantly. But it didn't eliminate it.

The Personas Are the Architecture

This is the part most people get wrong about AI personas. They treat them as role-play — "act like a senior engineer." That produces confident-sounding output with no actual depth. What works is what Brian Madison calls the BMAD method: personas aren't characters, they're skill compositions with documented authority boundaries. Each persona knows what it's responsible for, what it's allowed to decide, and — critically — where it must escalate to a human.

The most valuable persona I built was a clinical quality domain expert — grounded in the terminology, decision patterns, and judgment of real specialists in the domain. Not a generic "healthcare AI." A persona that knows the difference between a numerator exclusion and a denominator exception, that catches when a measure engine applies screening criteria to the wrong encounter type, that says "the specification changed in the latest update — your logic references the old version."

That domain expert persona exists because domain expertise is expensive to access. The real specialists have limited availability. Engineering decisions happen continuously, not on a meeting cadence. Most domain questions have clear answers an experienced specialist would give immediately — those should be available at machine speed. The hard questions — the ones requiring genuine clinical judgment — are identifiable by the persona and escalated. The result: faster routine decisions, better-quality escalation, and a documented reasoning chain the real experts can audit.

The other persona that changed how the system works is the Journalist — what we now call Quill. Someone has to tell the story. Not the code comments. Not the commit messages. The actual narrative of what happened, why decisions were made, what was tried and abandoned, what the system means. Without that, you accumulate what I call comprehension debt: the system works but nobody — including future-you — can explain why it works the way it does. Quill produces organizational memory that compounds instead of evaporating.

These aren't optional flourishes. They're structural. Remove the domain expert and the AI makes clinically incorrect decisions with high confidence. Remove the journalist and the project becomes opaque within weeks. The governance harness only works because the personas bring specific knowledge the AI doesn't have on its own.

The critical gap was always systems coherence verification. An AI that writes beautiful code to populate the results table, then forgets to refresh the materialized view the API reads from, has produced a system where the database says one thing and the screen says another. No persona review catches that — it only manifests at runtime, across service boundaries.

That's what loop engineering adds: the ability to say "after every pipeline run, verify that every downstream artifact matches the source of truth." Not as a suggestion. As an automated gate that fails loudly when things drift.

The Timing Question

When do you install these gates?

Don't gate during discovery. When you're building 8 measure engines in an afternoon, changing the API contract twice, pivoting the UI based on live feedback — tests are drag. You need flow state. The human IS the verifier.

Gate when interfaces solidify. Once the same bug bites you twice. Once the pipeline steps are stable. That's when you know the interface has settled enough to be worth encoding. The signal is: it broke because of drift, not change.

Gate what repeats. If you run it once, verify manually. If you run it daily for hundreds of clients, gate it. Verifier construction cost ÷ future executions = ROI. For hundreds of executions per year, even expensive tests pay for themselves in a month.

The Merged Model

Neither stage-gates alone nor loops alone are sufficient.

Stage-gates without loops: every mechanical task waits for human approval. Slow.

Loops without gates: everything runs autonomously, including compliance logic. Dangerous.

The merged system gives each task its correct altitude:

• "Implement a clinical quality measure with proper score extraction" → Human defines goal. Gate verifies compliance. Loop handles coding and testing.
• "Is our pipeline too slow for large orgs?" → Human flags it. Gate investigates architecture. Loop benchmarks and optimizes.
• "Refresh all materialized views after pipeline" → Pure loop. No human needed. Verifier: row counts match source tables.

Most teams try to put everything in one layer. Either everything needs approval (slow) or everything runs free (dangerous). The sweet spot is knowing which layer each task belongs in.

Five Rules for Production Loops

1. The verifier must be harder than the task. If your test is easier to pass than the real requirement, the loop finds the easiest path — not the correct one.

2. State belongs on disk. Context windows degrade. Sessions end. Models forget. If a loop needs to remember something, write it to a file.

3. Separate generator from verifier. The thing that writes code cannot judge code.

4. Hard stop at N retries. No loop runs forever. Without limits, you get infinite token burn on unsolvable problems.

5. Promote successful patterns. When a loop solves a problem, extract the pattern. The first time costs tokens. The 220th time costs nothing.

Cost Control

Three mechanisms:

Tier the loops. Pre-computed checks (row counts, schema matches) cost zero tokens. Parameterized SQL verifiers cost zero tokens. Only truly novel verification needs AI — rare and cached.

Invest in the verifier, not the loop. A good test suite runs in 10 seconds and catches 90% of problems. Cheaper than an AI retrying 5 times at 10,000 tokens each.

Promote aggressively. Every time a loop discovers something (like "scores are extractable from clinical notes with a simple pattern"), promote it to a template. First client costs tokens. The 220th costs nothing.

The Uncomfortable Truth

Here's what nobody in the loop engineering hype wants to say: most software problems don't have clean verifiers.

Training loss is numeric. Test pass/fail is binary. But "Is this measure engine CMS-compliant?" requires a human expert with 10 years of MIPS knowledge. "Is this UI clear to a quality administrator?" requires watching someone use it. "Will this scale to thousands of clients?" requires production traffic.

Loop engineering works brilliantly for the 60% that's mechanical. For the other 40% — design, compliance, judgment calls — you still need gates. You still need the uncomfortable conversation where someone says "this isn't right."

The teams that pretend everything can be looped will ship fast and break things that matter. The teams that refuse to loop anything will be too slow to compete.

The sweet spot is knowing the difference.

The Stale Baseline Trap

Here's something the academic papers don't mention: verifiers can be correct AND misleading.

We built a verifier that compared new pipeline output against a legacy results table — hundreds of measures, exact match required. Sounds rigorous. And it was — until the verifier itself became the problem.

The legacy table contained data from a prior pipeline run that used different engine routing rules. When we changed which engines handled which measures, the new pipeline correctly produced different numbers. But the verifier saw "different" and said FAIL.

A subset of measures showed discrepancies. We spent time investigating before realizing: the new output was right. The baseline was stale. The verifier was testing against yesterday's truth, not today's.

The lesson: A comparison verifier is only as good as its reference data. If the reference was produced by a different system configuration, you're not testing correctness — you're testing backward compatibility. Those are different things.

Three fixes: 1. Regenerate the baseline fresh before every comparison (expensive but honest) 2. Make the verifier self-contained: write data through the new path, read it back, verify the round-trip (no external reference) 3. Accept a documented divergence threshold when configuration has explicitly changed — but NEVER normalize away unexpected differences without root-cause

The dangerous mode is when the team learns to say "oh, that's just the stale baseline again" and stops investigating. One day it won't be the baseline. It'll be a real bug wearing the same clothes.

What We Got Wrong About Effort Estimation

The B-Team adversarial review estimated 3-6 weeks for the bitmap pipeline rewrite. The actual implementation took 8 hours.

This isn't because the B-Team was incompetent. It's because their estimation model assumed traditional execution: one developer, switching between files, running manual tests, waiting for code review, writing documentation. That's how the 3-6 week number makes sense.

But looped execution changes the math: - The pipeline had a single shared output interface. Changing a method signature was one addition plus a find-and-replace across dozens of call sites. That's minutes of mechanical work, not days. - The verifier ran immediately after each change — no "wait for QA" cycle. - The AI held full context (all call sites, the base class, the verifier, the state file) simultaneously. No context-switching tax.

The calibration insight: If your adversarial review estimates effort for a looped task, ask the reviewer to identify the shared abstraction boundary. If there IS one, the actual effort is the boundary change + repetition count × cost-per-repetition. The repetition cost for a find-and-replace is ~zero. The total effort is dominated by the boundary change alone.

Our B-Team now knows to ask: "Is there a single interface point?" before estimating. If yes, divide the estimate by the number of consumers — they're not independent tasks.

The Bottom Line

Loop the mechanical. Gate the judgment. Invest in honest verifiers. And test your verifiers against stale data — because a verifier that cries wolf trains you to ignore it.

The teams that get this right won't just ship faster — they'll ship with confidence. And in healthcare, where a wrong number can cost a practice $75,000 in MIPS penalties, confidence isn't the product. Correctness is. Confidence is what you earn when the verifier agrees.

Douglas Jones is an engineering leader and founder of Codifide, focused on AI-native software development with stage-gate governance in regulated domains.

The Agentic Stage-Gate Governance framework is open source.

References

1. Karpathy, A. (2026). AutoResearch. github.com/karpathy/autoresearch
2. Fortune (2026). "'The Karpathy Loop': 700 experiments, 2 days, and a glimpse of where AI is heading." fortune.com
3. Bilevel Autoresearch: Meta-Autoresearching Itself (2026). arXiv:2603.23420
4. Cooper, R.G. (1986). Winning at New Products: Accelerating the Process from Idea to Launch. Stage-Gate® methodology.
5. CMS MIPS Payment Adjustment. 42 CFR § 414.1405. Up to ±9% Medicare Part B adjustment based on composite performance score.
6. AI Builder Club (2026). "Karpathy's LOOPS.md: The Rules and What's Verified." aibuilderclub.com

Appendix: Stage-Gate vs Loop Engineering — Side-by-Side

Stage-Gate Strengths & Weaknesses

Loop Engineering Strengths & Weaknesses

How Merging Fixes Both

Decision Framework

Three questions:

1. Can a machine verify the output? → Yes: loop it. No: gate it.
2. Is the cost of being wrong catastrophic? → Yes: gate it, even if verifiable.
3. Will this task repeat? → Yes: loop it (amortize verifier cost).

The Numbers

• Hundreds of measures × hundreds of clients × quarterly refreshes = hundreds of thousands of loop-cycles/year (automated)
• Scaling to: hundreds of measures × thousands of clients × millions of patients = millions of loop-cycles/year
• 7 gates × 6 initiatives/year = 42 gated decisions/year (human judgment)
• At scale ratio: 41,800 automated executions per human decision

That's the leverage.

Related Articles

• Agentic Stage-Gate-Loop Whitepaper — the full framework specification
• AI Security Changed in 90 Days. Most SDLCs Didn't. — why traditional SDLC security breaks at agent speed
• Agentic SDLC Governance — checks and balances for AI-native software development
• AI Coding Agent Governance — authority boundaries and verification when agents write production code
• Building a Regulated Healthcare Platform — production case study applying these patterns