Cristian Ducu

When the benchmark was broken

The last post ended at a comfortable place: a best-of-N run, recorded as it happened, exported as a portable receipt that anyone can verify offline. Status: CLEAN. Most tools stop there, because the happy path is where demos live.

The interesting question starts after that. Six weeks later you discover the fitness harness was measuring the wrong thing. The evaluation your winning selection rested on is wrong. Every system you have ever used offers the same two options: quietly fix the data, or quietly ignore it. Both are lies of omission, and both are invisible to anyone who trusts your records.

Bellbook 0.5 gives you a third option, and it is the reason the project exists: say so, on the record, in a way that cannot be quietly undone.

Retract it

The author of the wrong evaluation retracts it. From Python:

r = w.retract(author="evaluator", target=evals[1],
              reason="fitness harness measured the wrong thing")
assert r.accepted

report = bellbook.validate(w.receipt())
print(report.status)                 # "tainted"
print(report.standing["unsound"])    # the selection that rested on it

Or from the CLI, if you never touched a language binding:

bellbook retract --log $LOG --rules $RULES --author evaluator \
  --target $e1 --reason "fitness harness measured the wrong thing"

bellbook export --log $LOG --rules $RULES --out receipt.json
bellbook validate receipt.json          # -> TAINTED, exit 2

The retracted evaluation does not disappear. It stays in the log, hash-chained like everything else; what changes is its standing. The receipt flips to Tainted, the selection that used the evaluation is marked unsound, and every candidate that descended from that selection is marked compromised, at any depth. You do not get to choose how far the damage report reaches. Replay computes it.

One subtlety the model gets right: a repair that was motivated by the broken evaluation, but derives from a still-sound baseline, stays sound. Intent is not contamination. The record distinguishes “this work rested on the lie” from “this work was prompted by discovering the lie”, because those are different things and a trustworthy record has to know the difference.

Who gets to retract

Retraction is ownership-bound. You may retract your own records; declaring someone else’s record wrong is contrary evidence, not a retraction. The escape hatch is explicit and in the trust policy where a verifier can see it: an admin retraction actor, named in the rules that every receipt embeds.

bellbook rules init --author agent:provider --author evaluator:provider \
  --author human:user --admin human --out rules.json

So a human principal can retract across authors, and anyone verifying your receipt can see exactly who held that power. Nothing about the arrangement is out of band.

Repair it, on the record

Recovery is one selection. Re-evaluate on something you still trust, then reaffirm the choice, naming the selection it replaces:

e_new = w.evaluate(author="evaluator", candidate=winner,
                   criterion="manual-review", passed=True)
s_new = w.select(author="agent", objective="best-of-n",
                 consider=cands, choose=[winner], uses_eval=[e_new.id],
                 replaces=s.id)

report = bellbook.validate(w.receipt())
print(report.status)                     # still "tainted" - permanently
print(report.standing["restorations"])   # {unsound id: [reaffirming id]}

The compromised line clears. The standing section records the restoration. And the receipt is still Tainted.

The part that matters

That last line is not a limitation. It is the whole point, so let me say it plainly:

Restoration restores standing, not Clean.

The receipt stays Tainted, exit code 2, forever, because the retraction is part of history. What the reaffirmation changes is the standing section: the line of work is recorded as restored, on fresh evidence, with the whole episode - the break, the reach of the damage, and the repair - permanently on the record.

Think about what the alternative would mean. A record that could quietly return to Clean after a retraction would be a record where “Clean” tells you nothing, because you could never know whether it was Clean the honest way or Clean the laundered way. The value of Clean is exactly that it cannot be re-earned by cleanup. And the value of a restored-but-Tainted receipt is that it tells the truth in both directions: something was wrong here, and here is the evidence it was handled.

This is what I mean when I say Bellbook records the consequences of intelligence, not its architecture. Agents will be wrong. Benchmarks will be broken. Evaluations will be retracted. A version-control system for autonomously evolving software has to represent being wrong as a first-class, verifiable event, or it is only a version-control system for the runs that went well.

What did not change

Bellbook 0.5.0 adds no record kinds and no spec change; the epoch stays 0.3, and every 0.3/0.4 receipt validates identically. The retraction machinery has been in the kernel since v0.2. What this release changes is reach: previously you needed the Rust core to tell this story; now the CLI and the Python package tell all of it, and the whole sequence above runs in CI on every push, on both surfaces, as the release gate. The honest path is not a footnote; it is the tested path.

Try it

cargo add bellbook          # Rust
pip install bellbook        # Python, same core, prebuilt wheels
  • The quickstart now has a Phase 2: it walks exactly this story, CLI and Python side by side.
  • cargo run --example broken_benchmark shows the deep version: a three-generation line collapsing and recovering.

It is an early release, not a settled standard. If you run a workflow through it and the model bends somewhere it should not, that is exactly the feedback worth having.