AutoAgent + Meta-Harness: The Agents That Build Better Agents

TL;DR: AutoAgent (2.2K stars) lets a meta-agent iteratively improve an agent's code overnight — score-driven hill climbing for agent architecture. Stanford's Meta-Harness used this pattern to discover an agent that hits 76.4% on Terminal-Bench 2.0 with Claude Opus 4.6. The key trick the evolution found: environment bootstrapping (snapshot the sandbox before starting, save 2-5 exploration turns). Hand-designing agent scaffolds is over.

The Manual Agent Engineering Problem

I have spent more hours than I care to admit doing the same thing: open the system prompt, change three words, adjust a tool definition, re-run the benchmark, stare at the score, decide if it went up or down, repeat. This is the life of anyone building agent harnesses right now.

Every cycle looks identical. You run your agent against a task suite. Something fails. You read the logs — all of them, line by line — trying to figure out where the reasoning went sideways. Was it the system prompt that didn't constrain the tool selection tightly enough? Was it the routing logic that sent a retrieval query to the wrong handler? Was it the output parser choking on an edge case? You form a hypothesis, edit the code, and run it again. If you're lucky, the score ticks up a tenth of a point. If you're unlucky, you broke something else.

This is bilevel optimization with a human in the inner loop. The outer loop is the benchmark score — the objective function you're trying to maximize. The inner loop is the agent architecture itself — the system prompt, the tool definitions, the orchestration logic, the retry strategies. And that inner loop is frozen at design time. Every improvement requires a human reading traces, identifying bottlenecks, and writing new code. The mechanism cannot improve itself.

Qu and Lu nailed this in their bilevel autoresearch framework: the mechanism matters more than the parameters. You can tune hyperparameters all day, but if the underlying mechanism — the way the agent gathers evidence, selects tools, routes between sub-agents — is suboptimal, no amount of parameter tuning will save you.

So the question becomes obvious: what if the agent could improve its own harness?

AutoAgent: Overnight Agent Evolution

This is exactly what AutoAgent does. Built by thirdlayer.inc, sitting at 2.2K stars on GitHub, MIT licensed, and deceptively simple in architecture.

The core design is a single-file agent.py that serves as the edit surface. This is the file the meta-agent is allowed to modify. Everything the agent does — its system prompt, tool definitions, orchestration logic, retry strategies, output formatting — lives in this one file. The human provides direction through program.md, a Markdown file where you describe what you want the agent to do, what behaviors to prioritize, what constraints to respect. Harbor benchmarks provide the scoring function, returning values between 0.0 and 1.0. And the whole thing runs inside Docker for isolation, so a bad mutation can't trash your system.

The loop is elegant. Run the agent against the benchmark. Capture the score. The meta-agent reads the agent's code, the benchmark results, and the execution traces. It identifies what went wrong — which tasks failed, which tool calls were suboptimal, where the reasoning chain broke down. Then it modifies the agent's code: rewrites a section of the system prompt, adds a new tool, changes the orchestration flow, adjusts the error handling. The modified agent runs against the benchmark again. If the score goes up, the change is kept. If it goes down, the change is discarded. Hill climbing, applied to agent architecture.

The human steers via Markdown. The meta-agent writes Python. You describe the direction in natural language — "focus on improving file navigation tasks" or "reduce hallucination in code generation" — and the meta-agent translates that intent into concrete code changes, validated against the objective function.

The pitch is simple: give it a benchmark, let it run overnight, wake up to a better agent. And the remarkable thing is that it actually works.

Meta-Harness: Proof It Works at 76.4%

The Stanford IRIS Lab took this pattern and pushed it to its logical conclusion with Meta-Harness. The repository has 597 stars and a result that demands attention: 76.4% overall on Terminal-Bench 2.0 — 89 tasks, 5 trials each, running Claude Opus 4.6.

Break that down by difficulty: Easy tasks hit 100%. Medium tasks land at 81.1%. Hard tasks — the ones that trip up most hand-designed agents — reach 64.7%. These are not trivial benchmarks. Terminal-Bench 2.0 tests real terminal operations: file manipulation, process management, network configuration, system administration tasks that require multi-step reasoning and tool use.

Here's the part that matters most: the agent architecture was discovered through automated harness evolution, not hand-designed by researchers. The Stanford team didn't sit in a room whiteboarding the optimal agent scaffold. They set up the meta-optimization loop, defined the objective function, and let the system search the space of possible agent architectures.

The architecture builds on Terminus-KIRA from KRAFTON AI and Harbor's Terminus-2 infrastructure. But what makes Meta-Harness distinctive is what the automated evolution discovered on its own.

The key innovation: environment bootstrapping. Before the agent enters its main task-solving loop, it first snapshots the sandbox environment. Working directory structure, available files, installed languages and runtimes, accessible tools, package managers, environment variables — all of it captured and injected as context before the agent takes its first action on the actual task.

This saves 2 to 5 early exploration turns that agents typically waste on orientation. Instead of the agent running ls, then which python, then cat /etc/os-release, then pip list — burning tokens and turns just figuring out where it is — the bootstrapping phase front-loads all of that information. The agent starts its first real turn already knowing what it has to work with.

This is a trick a human engineer might eventually think of. I've done something similar in my own harnesses — injecting system context into the prompt. But I arrived at it through intuition and trial-and-error over weeks. The meta-optimization found it automatically, as a natural consequence of score-driven search over the architecture space. The system discovered that agents perform better when they know their environment upfront, and it wrote the code to make that happen.

That's the difference between human-designed and evolution-discovered architecture. A human finds one good trick after weeks of debugging. An automated meta-agent searches the space systematically and finds tricks that humans might not think of for months — or ever.

What This Means for Us

This connects directly to what we're building with the investigate tool and the autonomous research lab.

Our current setup is the inner loop. The investigate tool runs adversarial analysis — cross-referencing claims against sources, identifying contradictions, scoring confidence levels. Quality gates filter the output. It works. But the mechanism is frozen. If the investigation strategy has a blind spot — say, it consistently under-weights certain source types, or it doesn't probe deeply enough on financial claims — the only way to fix it is for a human to read the investigation reports, identify the pattern, and manually update the evidence-gathering logic.

What we're missing is the outer loop. A meta-agent that reads investigation patterns across runs, identifies systematic failures in evidence gathering, and writes new investigation strategies. Not tweaking parameters — writing new mechanisms. New ways to decompose claims. New source-selection heuristics. New confidence-scoring rubrics. All validated against a scoring function that measures investigation quality.

The AutoAgent pattern is the template. Wrap our investigation tools in score-driven evaluation. Define what a good investigation looks like — completeness of evidence, accuracy of confidence scores, detection rate for planted contradictions. Let the meta-agent discover better investigation mechanisms overnight.

The environment bootstrapping insight from Meta-Harness applies directly. Before any investigation begins, snapshot what's available: which APIs are accessible, which databases can be queried, which search tools are online, what rate limits apply, what the current date and news cycle context looks like. Inject all of that as context before the first evidence-gathering step. Our investigations currently waste early turns on capability discovery — figuring out what tools are available and what they can do. Front-loading that information would tighten every investigation run.

The bilevel autoresearch paper from Qu and Lu showed that mechanism matters more than parameters. AutoAgent and Meta-Harness prove that this principle works for agent systems specifically. You don't need a human in the inner loop. You need a scoring function and a meta-agent with permission to edit the code.

The trajectory is clear. Hand-designing agent scaffolds — the system prompts, the tool routing, the orchestration logic — is becoming the new manual feature engineering. It works, it's what we all do today, and it's about to be automated away. The agents that build better agents are here. The only question is whether you're going to keep hand-tuning your harnesses or let the meta-agent do it while you sleep.

I know which one I'm choosing.