Version Your Judge First: Notes from Building a Self-Evolution Engine

This is the first entry in a new series: Self-Evolution Notes: field notes from building a self-evolution engine inside ROS. The premise of the engine is one sentence: humans set the goal, the system finds the path. Humans define the direction of improvement and the constraints; the engine generates scenarios, runs tests, scores results, proposes changes, A/B tests them, and deploys only what survives a significance test.

Every entry in this series follows the same format: the problem, how we found it, how we fixed it, and how the fix became leverage. No concept tours. Only bugs we hit and climbed out of. Anything with a number in it is reproducible.

What we are building

The ROS Self-Evolution Engine closes a loop: a scenario generator (test scenarios mined from real conversations) → a test runner (multi-turn simulated dialogues) → an LLM judge (scores each response on explicit dimensions) → an improvement generator (produces a new prompt version from the scores) → an A/B tester (Welch's t-test, p < 0.05 with effect size > 0.2 or it does not count) → auto-deployment (replace only on a significant win, always keep the last known-good version one rollback away).

This is not a new idea, and we do not pretend it is. Line up the last two years of work: the Gödel Machine (Schmidhuber, 2007) proposed provably beneficial self-modification, but required the system to prove each change was an improvement before adopting it: impractical in the real world. The Darwin Gödel Machine (arXiv:2505.22954, UBC × Sakana AI, ICLR 2026) replaced proof with empirical validation and improved itself from 20.0% to 50.0% on SWE-bench and from 14.2% to 30.7% on Polyglot. SICA (arXiv:2504.15228, NeurIPS 2025) lets a coding agent edit its own codebase and improved from 17% to 53% on a random subset of SWE-bench Verified.

These systems share one premise, and it is stated in their papers: all of them rely on an evaluator to answer "is this version actually better?" DGM leans on coding benchmarks. SICA leans on a utility function (0.5 normalized benchmark + 0.25 cost + 0.25 wall-clock time). Our engine leans on an LLM judge. If the evaluator says better, the system promotes the new version. That premise is where we got burned.

If your judge drifts, the steering wheel of the entire improvement loop is loose. That is the first lesson, and it is the whole of this note.

The problem: judges drift, and they are not spectators

The first piece of the engine we shipped is the evaluation layer: a tech radar that ingests external signals (papers, releases, benchmarks) plus internal review items, and has an LLM judge score each item on two separate axes: truthfulness and value, each with levels. The verdicts decide what makes the weekly report, what goes to archive, and what gets rejected.

In the first version, the judge was a prompt with no version management. After a few weeks, three symptoms showed up. Each one looks minor alone; together they are one disease:

• Re-runs disagree. Same snapshot, same prompt, different verdict on different days. An item scored L3 on Monday came back L2 on Thursday. For a self-improving system, "same input must produce the same output" is not a nicety: without it, the improvement signal is the noise;
• Rankings drift. With no change in content, the relative order of a batch moved. The top of that list feeds the weekly digest, so the selection criterion itself was drifting;
• "Who judged this?" was unanswerable. Whether a verdict came from the LLM or a human confirmation, and under which rule version, could only be guessed. A system that cannot answer that cannot review itself, and an improvement engine that cannot review itself cannot improve.

How we found it: three signals and one reframe

The drift was not caught once. It was the overlap of three signals: inconsistent re-runs, rankings that moved with the date, and a review meeting where we could not justify a verdict. The decisive move was to stop treating the judge as a tool and start treating it as an artifact:

1. Reproducibility became acceptance criterion #1. Same input + same judge version, three consecutive runs, identical verdicts. If that fails, no other metric matters;
2. Split "what is judged" from "who judged". Content, rule version, and verdict source became explicit fields: not context to be inferred;
3. Audit how the judge reads. We found the judge had been reading a truncated excerpt instead of the full snapshot. Change the truncation, and the judge's input changes. That is a physical source of drift.

How we fixed it: three anti-drift rules

1. Version the judge (judge_version). The rulebook is a versioned artifact. Same content + same version must produce the same verdict: enforced as a test assertion, not a convention. Our version went from q7-2026-09-08.1 to t2a-2026-09-08.2, each step with a regression suite;
2. Weights change only in weekly batches. No same-day weight tuning in daily runs. Weights are the slow variable; they move in the weekly batch, so "today's mood" never leaks into a verdict;
3. Feedback tunes weights, never gates. Truthfulness and value stay strictly separated. "Not useful" (a value judgment) can never imply "unsourced" (a truthfulness downgrade). Click feedback may move weights; the four truthfulness gates do not accept feedback at all: low engagement must never launder "no source" into "has source."

Supporting engineering: verdict source became a field (pending | llm | human); internal review items render in a separate section instead of mixing with external signals; migrations refuse to overwrite artifacts from a different judge_version; the self-test suite grew from 27 to 40 cases, including anti-hallucination assertions like "an empty quote must fail."

How it became leverage: from incident to discipline

The real output of this incident is not "it is fixed." It is four reusable things:

1. The version number became a field. judge_version travels with every verdict, so any historical verdict can answer "which rules judged this?";
2. "Who judged" became a field. Three-valued source, so LLM verdicts and human confirmations are different things in the data layer, and render differently too;
3. The anti-drift rules live inside the judge. The three rules are part of the judge prompt itself and get re-reviewed with every version: the artifact being evolved carries the rules of its own evolution;
4. A design principle. The evaluator is a first-class citizen of the evolution engine: it must itself be versioned and guarded by regression tests. Our engine doc says "the judge's prompt is evolvable too." This incident added the second half of that sentence: evolvable is not the same as drift-prone. Before you evolve your evaluator, make it stable.

One byproduct, from cost accounting: a submission tried to offer "4×–24× faster than the previous version" as cost evidence. Rejected. A relative speedup cannot be converted into an absolute hard line like "p95 ≤ 6000 tokens": only absolute numbers count as cost data. That ruling is now part of the rule version, so it will not have to be re-litigated next round.

Three things you can take away

If you are building any "AI evaluates AI" loop: automated scoring, model judges, self-improvement pipelines: here is the part that transfers:

1. Version your judge before you automate evolution. An improvement loop on an unstable evaluator compounds noise, not capability;
2. Make "who judged, under which rules" a field, not something reconstructed from memory and context;
3. Keep feedback away from the gates. Improvement signals may come from user behavior; quality floors do not accept votes.

One last thing, because it's the reason we started: everyone is unique, and a generic memory cannot solve everyone's problems: because "what is good for you" is a judgment that cannot be outsourced to an average model. The engine is the same principle applied to the system itself: "what counts as better" is a judgment that cannot be outsourced to a judge that drifts. Next entry: what actually differs, in engineering terms, between a generic memory and a person-centered one.

References

1. J. Schmidhuber. Gödel Machine: A Self-Referential Problem Solver, 2007 (theoretical framework for provably beneficial self-modification).
2. J. Zhang, S. Hu, C. Lu, R. Lange, J. Clune. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents. arXiv:2505.22954, 2025 (ICLR 2026). SWE-bench 20.0%→50.0%, Polyglot 14.2%→30.7%.
3. M. Robeyns, M. Szummer, L. Aitchison. A Self-Improving Coding Agent. arXiv:2504.15228, 2025 (NeurIPS 2025). 17%→53% on a random subset of SWE-bench Verified.
4. N. Shinn et al. Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366, 2023.
5. E. Zelikman et al. Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation. arXiv:2310.02304, 2023.