A verifiable receipt for your best-of-N run

Most agent setups already run some form of best-of-N. Generate a few candidate changes, score each one, keep the winner. It works. But six months later, when someone asks which candidate shipped and on what evidence, the honest answer is usually a shrug and a log file that could have been edited an hour ago.

Bellbook 0.4 closes that gap without changing how you run the loop. You keep generating and scoring candidates exactly as you do now. Bellbook records what was considered, the evaluation evidence behind it, and which one you chose, into an append-only, hash-chained log that anyone can replay and verify offline. As of this release the whole loop runs from the command line alone, no Rust or Python binding required.

The loop you already run

Three record kinds map onto best-of-N directly:

The decision stays yours. Bellbook never ranks or scores for you. It records the choice and makes it impossible to quietly rewrite later.

Recording it

Verification is always relative to a rules file: the trust policy that says which actor ids may author records, what evidence counts, and so on. It is embedded in every receipt, so a verifier re-derives every decision against it rather than trusting your say-so. Generate a starter and bind each actor to a role:

bellbook rules init --author agent:provider --author evaluator:provider --out rules.json

Then record the run. Each command prints the committed record id:

RULES=rules.json
LOG=./mylog

c0=$(bellbook candidate add --log $LOG --rules $RULES --author agent \
       --git-tree a1b2... --json | jq -r .id)
c1=$(bellbook candidate add --log $LOG --rules $RULES --author agent \
       --git-tree c3d4... --json | jq -r .id)

e0=$(bellbook eval add --log $LOG --rules $RULES --author evaluator \
       --candidate $c0 --criterion fitness --score 40 --scale 0 --json | jq -r .id)
e1=$(bellbook eval add --log $LOG --rules $RULES --author evaluator \
       --candidate $c1 --criterion fitness --score 90 --scale 0 --json | jq -r .id)

# choose the winner; name the evaluations the choice rests on
bellbook select --log $LOG --rules $RULES --author agent --objective best-of-n \
  --consider $c0 $c1 --choose $c1 --uses-eval $e0 $e1

If you would rather stay in Python, the same run is a few lines against bellbook.Writer, and bellbook.default_rules({"agent": "provider", ...}) builds the rules object inline so you never hand-author one. Same core, same decision.

The receipt

Bundle the log into a portable receipt and verify it. This is the part that did not exist from the CLI before 0.4:

bellbook export --log $LOG --rules $RULES --out receipt.json
bellbook validate receipt.json          # -> CLEAN

receipt.json is self-contained. Hand it to anyone. bellbook validate recomputes every record id, walks the hash chain from genesis, re-derives every verdict, checks signatures and evidence, and reports Clean, Invalid, or Tainted with an exit code you can gate CI on. Edit one byte of the receipt and every id below it breaks. No access to your harness, your machine, or a network is needed to check it.

Where it earns its keep: a benchmark that was wrong

The interesting case is not the happy path. It is the day you discover a metric was broken and a decision rested on it.

Retract that evaluation and replay. Bellbook marks every candidate that depended on it compromised, at any depth in the lineage, not just the direct child. The receipt goes from Clean to Tainted, and the taint is specific: it names exactly which lines of work are affected and which are untouched. Nothing is silently dropped, because in Bellbook the absence of a record is itself data.

Recovery is one record. Make a fresh selection on evidence that survived, and replay restores the line, with the whole episode, the break and the repair, permanently on the record. You are never asked to trust that someone cleaned it up. You can see it.

What Clean means, and what it does not

It is worth being precise, because the value is in the boundary:

• Consistency, not completeness. A Clean receipt means the captured history is internally consistent under its embedded rules. Whether everything the agent did was captured depends on how your host instruments its runtime. Bellbook proves the record is intact and honestly graded, not that the record is whole.
• Integrity, not confidentiality. Records and receipts carry full payloads in the clear. A receipt inherits the sensitivity of everything in it, so never put secrets in records and treat sharing a receipt as disclosure.
• Comparable only against rules you trust. Clean is relative to the embedded rules. Compare rules_hash against a policy you trust before relying on someone else’s receipt.

That honesty is the point. Bellbook is an evidence layer, not a logger, a database, or a runtime. It answers one question well: can I prove this record of what happened has not been altered, and that its decisions follow from its own stated evidence?

Try it

cargo add bellbook          # Rust
pip install bellbook        # Python, same core, prebuilt wheels

• Quickstart: the full best-of-N walkthrough, CLI and Python side by side.

• Worked examples in the repo: iterative_evolution (a multi-generation loop), repair_reevaluate (why a repair motivated by a broken evaluation is not tainted by it), and broken_benchmark (the compromise a broken metric casts, and one-record recovery).

• GitHub repository

• Rust crate

• Python package

• API documentation

• Specification

It is an early release, not a settled standard. If you put it in front of a real workflow and it bends in a way it should not, that is exactly the feedback worth having.