I Built Software That Improves Itself While I Sleep — 551 Experiments, 72 Improvements

The short version: What if your software could improve itself? Not in theory — actually run experiments, measure results, keep what works, revert what doesn't. Autonomously. While you sleep. I built a system that does exactly this. It's been running for months, has executed 551 experiments, and kept 72 improvements without a single human edit to the agent-owned files. Here is the architecture.

The concept: Karpathy-style AutoResearch

The idea comes from Andrej Karpathy's framing of autonomous research loops: measure the current state, hypothesize an improvement, implement it, measure again, keep or revert.

Measure baseline → Generate hypothesis → Implement change → Measure again → Keep or revert → Log everything

The key insight is that AI models are now good enough to generate meaningful hypotheses about code improvements. Combine that with automated measurement and you get a system that genuinely evolves — not in a hand-wavy AGI sense, but in a strict measure-and-validate sense.

The architecture: 11 specialist agents

My system has 11 specialist agents, each responsible for one part of the codebase:

Each agent runs on a different frequency — high-impact agents (voice, knowledge-base) run every iteration, while stable systems (scraper, verifier) run every 10th. Tight scoping per agent is the difference between a system that improves and a system that thrashes.

The orchestrator

The orchestrator manages the cycle. It determines which agents run, spawns the hypothesis engine, collects results, and logs everything.

// orchestrator.js (simplified)
import { HypothesisEngine } from './hypothesis-engine.js';

const AGENT_SCHEDULE = {
  voice:            { every: 1 },
  'knowledge-base': { every: 1 },
  scoring:          { every: 10 },
  scraper:          { every: 10 },
  email:            { every: 6 },
  outreach:         { every: 4 },
  'call-queue':     { every: 2 },
  'lead-discovery': { every: 3 },
  docs:             { every: 5 },
};

async function runIteration(iterationNumber) {
  const agentsToRun = Object.entries(AGENT_SCHEDULE)
    .filter(([_, config]) => iterationNumber % config.every === 0)
    .map(([name]) => name);

  for (const agentName of agentsToRun) {
    const engine = new HypothesisEngine(agentName);
    const baseline = await engine.measureBaseline();
    const hypothesis = await engine.generateHypothesis(baseline);

    if (!hypothesis) continue;

    const backup = await engine.backupCurrentState();
    await engine.applyChange(hypothesis);
    const newScore = await engine.measureAfterChange();

    if (newScore > baseline.score) {
      await engine.logResult('kept', hypothesis, baseline.score, newScore);
    } else {
      await engine.revertTo(backup);
      await engine.logResult('reverted', hypothesis, baseline.score, newScore);
    }
  }
}

The hypothesis engine

This is where the actual reasoning happens. The hypothesis engine reads the agent's current code, its program file (data, feedback, experiment ideas), and its history of past experiments. It then generates a concrete change to try.

// hypothesis-engine.js (simplified)
export class HypothesisEngine {
  constructor(agentName) {
    this.agent = agentName;
    this.programFile = `programs/${agentName}.md`;
    this.resultsFile = `results/${agentName}.tsv`;
  }

  async generateHypothesis(baseline) {
    const program = await fs.readFile(this.programFile, 'utf-8');
    const history = await fs.readFile(this.resultsFile, 'utf-8');
    const currentCode = await this.readAgentCode();

    const response = await claude.messages.create({
      model: 'claude-opus-4-7',
      max_tokens: 4096,
      messages: [{
        role: 'user',
        content: `You are an AI research agent. Improve this system.

CURRENT CODE:
${currentCode}

PROGRAM (data, feedback, experiment ideas):
${program}

PAST EXPERIMENTS (what worked, what didn't):
${history}

CURRENT SCORE: ${baseline.score}

Generate ONE specific, testable hypothesis for improvement.
Return the exact code change to make as JSON:
{"id": "...", "description": "...", "old_string": "...", "new_string": "..."}
Be conservative — small changes that are likely to help.`
      }],
    });

    return this.parseHypothesis(response);
  }
}

The trick is forcing the model to output a verbatim diff (old_string / new_string) rather than a free-form description. The runtime then verifies old_string exists exactly in the file before applying. This catches hallucinated changes before they corrupt the codebase.

Program files: the human-AI interface

Each agent has a program file — a Markdown document containing data, feedback, and experiment ideas. This is how humans guide the system without manually editing code:

# Voice Agent Program

## Simulation Data
- Round 5 (30 calls): 9.1/10 avg score, zero compliance violations
- Industry coverage: 78+ industries tested
- Weakness found: agent talks too fast when handling objections

## Feedback from Real Calls
- Call #3 (Cafecito Lounge): Owner was confused by the opening line. Simpler intro needed.
- Call #4 (Eliyanna Cafe): Good conversation but agent didn't ask for email.

## Experiment Ideas
- Try shorter opening: "Hi, this is Maya from QuickComet. Quick question about your business website."
- Add a pause after the opening question to give them time to respond.
- Test whether mentioning "AI-powered" increases or decreases interest.

The agents read these program files and use the data to generate better hypotheses. You never edit the agent's code directly — you add data to the program file and let the system figure out how to use it. This is the "curate, don't code" pattern that makes the workflow scalable for a solo founder.

Results after 551 experiments

Total experiments:  551
Kept (improved):     72 (13.1%)
Reverted:           420 (76.2%)
Crashed:             59 (10.7%)

A 13% improvement rate might sound low, but consider: each "kept" experiment represents a genuine, measured improvement. After 72 successful changes, the system is dramatically better than where it started.

The voice agent's prompt went from 12,657 characters to 7,402 (41% compression) while scoring higher on quality metrics. The knowledge base grew from 10 industries to 35 with 21 FAQ handlers and 20 objection handlers — all added by the agents, not by humans.

Key lessons

1. Most hypotheses fail, and that's fine. A 13% success rate is excellent for autonomous experimentation. The system learns from failures by logging discarded hypotheses so future iterations don't retry similar paths.
2. Crash handling is essential. 10.7% of experiments crashed. The system must revert cleanly from any state — backup before applying, restore on failure, log the crash for human review.
3. Program files are the lever. The quality of experiments is directly proportional to the quality of data in program files. Feed the system real feedback and it generates better hypotheses. Feed it nothing and it generates filler.
4. Agent frequency matters. Running every agent every iteration wastes compute. Stable systems (scraper, verifier) need fewer iterations. High-impact systems (voice, KB) benefit from every run. We tune this per-agent in the schedule.
5. Claude Opus is the right model for hypothesis generation. Smaller models generate superficial changes (rename a variable, add a comment). Opus generates structurally meaningful improvements (compress this prompt by extracting shared rules into a section). Pay for Opus on hypothesis; use Sonnet on measurement.
6. Verbatim diff parsing is mandatory. If you let the model output free-form "here's how to fix it", you spend 80% of dev time chasing hallucinations. Force old_string/new_string JSON output and verify old_string exists before applying.

Related reading

The same hypothesis-generation pattern shows up in other production AI systems we ship — see Why I'm Still Building RAG in 2026 for how it applies to retrieval architecture decisions, and Why RAG Fails in Production for the specific failure modes the AutoResearch loop catches in RAG pipelines.

Frequently Asked Questions

Bottom line

Self-evolving software is not a future thing. The pattern works today, on real production systems, with current models. The blockers are not capability — they are discipline (small reversible diffs, automatic revert, tight per-agent scope) and metric design (you have to be able to measure improvement cheaply).

Building or planning to build an AutoResearch system for your own product? Email hello@quickcomet.com — happy to compare notes on architecture or measurement design.