When the Judge Can't Judge: U Is a Verdict, Not a Failure

Our self-evolution engine runs a loop: scenarios go in, tests run, an LLM judge scores the results, an improvement generator proposes a new prompt, an A/B test decides, and deployment happens only on a significant win. Every part of that loop can fail. This note is about the failure mode we designed for last: the judge itself.

Field note 01 was about a judge that drifts: same snapshot, same prompt, a different verdict on a different day. That note assumed the judge shows up. This one starts where the judge doesn't. What is a loop supposed to do when the component that decides "better" has no answer today?

The problem: three exits from a failing judge, and all of them lie

The first piece of the engine we shipped is the evaluation layer: a research radar that ingests external signals (papers, releases, benchmark results) plus internal review items, and asks a judge to score each one on two axes, truthfulness (A through D) and value (L0 through L4). The verdicts decide what makes the daily digest, what gets archived, and what gets rejected. The run is scheduled, the budget is 15 judged items a day, and the judge is one LLM call per item.

So the question arrived on day one, before any code: when the judge fails (timeout, quota, provider down), what happens? We walked the three obvious exits, and none of them survive contact with the data:

1. Crash. The run produces nothing. The pipeline's reliability is now the reliability of its least reliable component, and one 502 at 3 a.m. costs a day of digest;
2. Silent fallback. Rules mode quietly replaces LLM mode, and the row still says judge_mode=llm. Now the data claims a provenance it doesn't have. A row that reads judge_mode=llm, value_level=U looks like "the model judged this undetermined." The model never ran. Calibration, hit rate, and the digest's own selection all stand on that false premise;
3. Guess anyway. Force a verdict so the schema is satisfied. Once the schema demands a letter, the letter gets manufactured. "Nobody looked at this" and "looked, and it's bad" collapse into the same symbol, and no downstream consumer can tell them apart.

The common bug is not the judge failing. It's a schema with no way to say "nobody looked at this one." We needed a fourth exit, and it had to be a state, not an error path.

How we found it: the acceptance test came first, then a boring audit question

Two signals, in order.

The first one we forced on ourselves: we wrote the acceptance list before building the judge. One line on it: a stub LLM that raises on every call, our stand-in for a timeout, must produce U, and the rest of the pipeline must still complete. Writing that line before the feature is what pushed the design question into the open. If the middle of the pipeline can be missing, what does the pipeline owe the parts downstream? We couldn't answer that with a try/except, so it got a schema change instead.

The second signal arrived weeks later, and this one stings. We asked the boring audit question: has this judge ever actually called a model? The answer was no. All fifteen items from the first full run carried judge_mode=rules, value_level=U, q2_source=pending. The code contained no LLM client at all. The --llm flag exited with code 2 and a FATAL line. The gate module's build_llm() raised NotImplementedError by design, even with a 164-character OPENAI_API_KEY sitting right there in the environment, fully valid, completely unused. The decision was deliberate: no provider wired means no fake automation. Rules mode with U, rerun when a real client lands.

That audit also found a label bug. Eight rows carried judge_mode=human. They were written by a model in a working session: not a human annotator, not an API call. The label promised a provenance the data couldn't back, and nothing in the output would have told you. You cannot tell from a verdict whether a judge judged.

An older lesson resurfaced in the same review. A fetch failure had once been recorded as "no signal," a crawler fact wearing a judgment's clothes. fetch_status=failed is what happened; decision=reject is what we think it means. Those belong in different fields, and they do now.

How we fixed it: U is a state, and four rules keep it honest

U entered the schema as a first-class value: truth_level is A/B/C/D/U, value_level is L0 through L4 plus U. U means undetermined. Not bad. Not useful. A state, not an error. Then four rules keep it from rotting:

1. The degradation path is explicit and narrow. The LLM call raises or returns nothing, and the row gets truth_level=U with the reason "LLM judgment failed, undetermined, auto-rerun next day." What never touches the LLM: the crawler's fetch, dedup, snapshot, and ledger; the four mechanical truth gates; the Q3 lookup table. The daily run completes, and the U rows get picked up tomorrow. The loop degrades; it doesn't stop;
2. No silent degradation. Asking for LLM mode when it isn't wired is a hard failure with exit code 2: no key raises one error, a key without a client raises another. Never a stub that looks usable. The mode you asked for is the mode that runs, or the run stops;
3. Anti-hallucination stays separate from degradation. Every gate verdict carries a quote from the source snapshot, and an empty quote fails the gate. "The judge ran and found nothing" is D with a reason, not U. U is only for "the judge didn't run." Collapse those two and the pipeline learns to lie with a straight face;
4. U never enters the stats. The value-accuracy calculation excludes U from the numerator and the denominator alike. If every row is U, accuracy is None. Not 0%, not 100%. A metric that can't tell "no data" from "all wrong" is not a metric.

Two supporting pieces, both from note 01. Determinism is an assertion, not a hope: same input plus same judge_version, three consecutive runs, identical verdicts (t2a-2026-09-08.2 as of this writing). And artifacts from different judge versions refuse to overwrite each other, so yesterday's accepted run can't be quietly rewritten by today's rules. The daily digest prints the semantics at the top for the same reason: U means undetermined, not counted as useful, not in the accuracy number.

How it became leverage

The real output isn't "the crash is handled." It's four things that outlived the incident.

The self-test is the contract. Forty checks, and the suite's exit code is the verdict: Q3's four branches (including "a relative speedup is not cost data"), Q2's positive and negative cases, three-run determinism, empty-quote failure, LLM-exception-to-U, schema validation, and the gate tests: resolve() raises, build_llm() raises, rules mode records source=pending instead of pretending. Change the judge without the suite, and the suite says no.

U became a field, and the fields record provenance. judge_mode, q2_source, value_source, counted_in_weight_stats. "Who judged, under which rules" is now a query, not a recollection. The eight mislabeled rows are the reason those fields exist.

The fact/judgment split became a rule. Crawler facts (fetch status, error code, snapshot hash) never share a field with judge conclusions. One records what happened; the other records what we think it means. The day those merged was the day a failed fetch looked like a boring Tuesday.

The design principle. An evaluation layer you cannot audit is not an evaluation layer. And the corollary we keep for the engine: a component that must always produce a number will eventually produce a fake one. The honest "I don't know" is cheaper than the confident wrong answer, and it's the only thing that keeps the rest of the numbers worth reading.

The literature backs both halves of that. GPT-4 as a judge matches human preferences over 80% of the time, about the rate humans agree with each other (arXiv:2306.05685), yet swap the order of two answers and Vicuna-13B beats ChatGPT on 66 of 80 queries (arXiv:2305.17926). The agreement is real; so is the failure mode. The models also mostly know what they don't know when asked directly (arXiv:2207.05221), and abstaining on the few items where they don't is a net reliability win (arXiv:2404.10960). Our U is that abstention, moved out of the model and into the schema, where a test can see it.

One last thing, because it's the reason we started: everyone is unique, and "what is good for you" is a judgment that cannot be outsourced to an average model. The engine is the same principle pointed inward: "what counts as better" can't be outsourced to a judge that drifts (note 01) or to a judge that guesses when it can't see (this note). U is that principle with a schema attached. Next entry: the A/B tester's statistics. Why Welch's t-test, why p < 0.05 with effect size > 0.2, and what "no significant difference" actually does to the loop.

References

1. L. Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685, 2023. Strong LLM judges match human preferences over 80% of the time, about the same rate humans agree with each other; documents position, verbosity, and self-enhancement biases.
2. P. Wang et al. Large Language Models are not Fair Evaluators. arXiv:2305.17926, 2023 (ACL 2024). Swapping the order of two responses let Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT as the judge; proposes multiple-evidence and balanced-position calibration.
3. S. Kadavath et al. Language Models (Mostly) Know What They Know. arXiv:2207.05221, 2022. Larger models predict P(IK), the probability they know an answer, with calibration that improves with scale; weaker on unseen tasks.
4. C. Tomani et al. Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations. arXiv:2404.10960, 2024. Abstaining on a few highly uncertain samples improves correctness by 2 to 8 points and avoids about half of hallucinations on unanswerable questions.