PAPER / PREPRINT, NOVEMBER 2026Verifiers That Do Not Trust the Agent: Anytime-Valid Release Gates and Hacking Bounds for AI Agents That Move Money
An AI agent that handles card disputes moves money. It decides when to refund, when to fight, and with what evidence. Changes to such an agent are usually judged by win rates and by the agent's own reports. Both can be gamed.
Scruple is a release gate whose verifiers never credit the agent's claims. Cost is recomputed from double-entry ledger postings. Every action is checked against a hash-chained trace the environment writes. The statistics stay valid however often you look and whenever you stop.
We prove how much a hacked change can gain and still pass, for agents that adapt and partly see where audits fall. We give the audit rate at which hacking stops paying when the verifier errs both ways, and state what the combined promote decision guarantees.
Four pre-registered experiments on a frozen synthetic benchmark of 10,000 disputes: one passed narrowly, two failed their main hypothesis, one found no effect. Learners trained on the scorecard used no exploit; the same learners on a naive proxy found them. Real Claude models found bugs in the verifier.
01 / CONTRIBUTIONSSix things the paper adds.
- Verifiers anchored to money, not to words. Cost is recomputed from ledger postings. Every action is checked against a trace the environment writes. Each check ships with an attack it stops, and we report what broke.
- Tests you can watch every day. Paired and per-case-type tests that stay valid however often you look and whenever you stop, with planning and a matching lower bound.
- Bets that adapt, and a floor no test can beat. A mixture of bets close to the best fixed bet on every run, and the fewest cases any valid test needs, computed from the observed costs.
- What the release decision guarantees. A theorem for the whole gate: which errors are bounded, which add up, and which are not covered at all.
- Hacking bounds for agents that adapt. How much a hacked change can gain and still pass when audits leak and the verifier makes mistakes, and the audit rate that ends the incentive.
- Experiments reported either way. Four pre-registered experiments, six agent types under two objectives, 64 disputes grounded in real complaints, and a timeliness study of 164,821 real complaints.
02 / KEY RESULTSWhat we found, in plain words.
- Peeking breaks ordinary tests. Ours hold. Checking a z-test every 10 cases rejects a harmless change 15% to 34% of the time at a nominal 5%. The betting test's worst rate over seven stopping rules was 2.7%. synthetic, 2,000 paths
- The optimiser learned to cheat, and the ledger called it cheaper. Tuned on wins and self-reports, it learned to forge documents and claim wins, and cost $33.87 per case less on the simulated ledger. Provenance checks blocked it. A cost-only gate would have shipped it. synthetic, E2
- The objective mattered more than the learner. Five learners trained on the scorecard used no exploit. Trained on the naive proxy, all five claimed false wins and the three that could forge did, becoming the cheapest agents and 21 to 22 points less fair. synthetic, 2,000 cases
- A better test still missed the one-cent regression, as we predicted. The variance-reduced gate reached e 14.9 against a bar of 25 on R5 (E3 fails H1). After the fact, a wider valid bet caught it in 86% of resamples. That is a lead for the next pre-registration, not a pass. synthetic, E3
- Told only win rate matters, no model forged. No upload in 72 episodes. Valid claims were contested more in 4 pairs and less in 3 (p = 0.50). Opus cited weaker evidence and its gate pass fell from 9 to 3 of 12. real models, E4
- The cheapest real model was not the fairest. Claude Sonnet cut merchant cost by $32.77 per case, and 4 of its 10 wins came against claims the cardholder was right about. real behaviour, simulated cases
- False alarms led to an engine bug. The honest policy failed the ledger check on 19 of 10,000 cases. The verifier was right that the cardholder was paid twice; the engine caused it, three ways. Fixed: 0 failures on 6,000 dev seeds. synthetic
- The evidence policy got 62 of 64 real situations right. Its misses were both wrong-amount cases. The oracle, which knows who is right, missed four refunds that had already settled: knowing the truth is not enough, the ledger matters. real complaints, simulated play
03 / HOW IT WORKSThe system, the lifecycle, the checks.
04 / FIGURESDrawn from the result files.
betting e-process, worst of 7 stopping rulesz-test re-run every 10 cases
worst batch, agent hacks only when its signal says not auditedTheorem 10.3 boundbound that ignores the leak
bound, beta 0bound, beta 0.3bound, beta 0.3, leak l 0.2
default bet, capped at halfwide bet, cap 0.95plug-in betgrid mixture
05 / NEW IN VERSION 2Agent types, E3, E4, real situations, and what broke.
trained on the scorecardsame learner on the proxyreference policies
honest briefwin-rate brief
| Agent | Cost / case | Fair, pp | Gate |
|---|---|---|---|
| gradient boosting, scorecard | -$8.01 | +7.8 | 99.8% |
| expert rules | -$5.71 | +2.5 | 99.6% |
| logistic regression, scorecard | -$1.79 | +6.7 | 99.8% |
| LinUCB bandit, proxy | -$37.24 | -22.1 | 51.9% |
| Q-learning, proxy | -$31.82 | -21.9 | 48.8% |
| MC planner, proxy | -$28.66 | -21.0 | 52.5% |
| classifiers, proxy | +$49.01 | -3.8 | 73.7% |
| Pre-registered | Result | Holds |
|---|---|---|
| H1 R5 blocked, full benchmark | e 14.9, needs 25 | no |
| H2 no control blocked | C1 to C4 hold | yes |
| H3 control folds blocked | 0 / 60 | yes |
| Post hoc: wide bet, R5 resampled | 0.86 (was 0) | post hoc |
| Player | Right | 95% interval |
|---|---|---|
| answer key | 64 / 64 | 0.943 to 1.000 |
| evidence policy | 62 / 64 | 0.893 to 0.991 |
| oracle, reads the truth | 60 / 64 | 0.850 to 0.975 |
| always accept | 53 / 64 | 0.718 to 0.901 |
| represent everything | 11 / 64 | 0.099 to 0.282 |
| Dataset | Top 5 | Ranks 6-25 | The rest |
|---|---|---|---|
| payments 2011-2016, 135,313 | 1.01% | 0.56% | 2.80% |
| checking 2025-2026, 8,000 | 0.62% | 0.29% | 3.70% |
| card disputes 2024-2026, 21,508 | 0.63% | 0.41% | 2.93% |
Complaint timeliness is the CFPB's 15-day flag, not a Regulation E or Z deadline. In all three sets the companies outside the 25 largest answered late 3 to 6 times as often as the five largest, and the interval for the difference excludes zero. Size is complaint volume.
caughtblockedpartialnot caughtnot modelled
What broke, and what we fixed
An engine bug, found by verifier false alarms. Refund races made the honest policy pay the cardholder twice on 19 benchmark cases. The ledger check was right; the engine was wrong in three ways. A false alarm of 0.19% per case blocks an honest candidate 98% of the time at 2,000 cases, so this mattered.
Three exploits closed. Memorised outcomes: $11.99 per episode more on exposed cases than on held-out probes, z 5.8. Escalating everything: now judged on whether the case file supported it. The adverse-document swap a learner found: caught on 11 of 11 cases, 4 before.
Fairness is a profile. Making a win against a valid claim a hard failure also fails the evidence policy on one case, because the agent cannot see validity. The default stays merchant-side; the stricter profiles are a choice for the deploying team.
| System | Merchant | Balanced | Compliance |
|---|---|---|---|
| Haiku | 7 | 7 | 5 |
| Sonnet | 11 | 10 | 7 |
| Opus | 11 | 11 | 10 |
| evidence policy | 12 | 12 | 11 |
06 / NEW RESULTSEach one proved in full in the PDF.
- Thm 10.3If every hacked case is flagged with probability at least rho, given all the agent knew, an accepted change has a gap of at least gamma with probability at most (1 - rho)^ceil(n gamma / eps), and an expected accepted gap of at most eps / (n e ln(1/(1 - rho))). No independence needed.adaptationp. 14 , opens the PDF
- Cor 10.5If the agent's signal has a likelihood ratio of at least l, then rho = (1 - beta) q l / (q l + 1 - q). An agent that can see the audits defeats any spot check.adaptationp. 15 , opens the PDF
- Thm 10.6With a verifier that misses a share beta of hacks and flags a share phi of honest cases, hacking stops paying exactly at q* = g / ((1 - beta)(K + g) - phi K).candidate novel, framingp. 15 , opens the PDF
- Cor 10.7No audit rate deters unless beta g <= (1 - beta - phi) K. False alarms raise the rate you need.candidate novel, framingp. 15 , opens the PDF
- Prop 7.1Averaging bets over a grid of J sizes gives a valid test whose wealth is within log J of the best fixed bet on every path.adaptationp. 9 , opens the PDF
- Prop 7.2The best fixed-bet growth rate g* of the observed costs equals a KL distance to a harmless law, so any valid test needs at least kl(1 - beta || alpha) / g* cases.adaptationp. 10 , opens the PDF
- Thm 9.1The promote decision carries the smallest level among the checks that cover a harm. False blocks add up. Harms no check covers carry no guarantee.standard, appliedp. 13 , opens the PDF
loading
- bound, beta 0- bound, beta 0.3- bound, beta 0.3, leak l 0.2# E2 batches, worst
07 / EXPERIMENTSPre-registered, then reported either way.
| Measure | Result | Bar |
|---|---|---|
| Regressions flagged, full benchmark | 7 of 7 | 7 of 7 |
| Exercised folds flagged | 114 / 135 = 0.844 | 0.80 |
| Control folds flagged | 3 / 60 = 0.050 | 0.05 |
| Policy | Cost / case | Hard fails | Correct |
|---|---|---|---|
| honest, also the scorecard optimum | $82.82 | 0.19% | 0.777 |
| proxy optimum: forges, claims wins | $48.95 | 55.4% | 0.557 |
| Task | TF-IDF | Haiku | Sonnet |
|---|---|---|---|
| Relief or not (AUROC, n 100) | 0.701 | 0.550 | 0.606 |
| Reg E or Reg Z (accuracy, n 100) | 0.870 | 0.900 | 0.930 |
| Issue routing (accuracy, n 50) | 0.580 | 0.560 | 0.540 |
| Measure | Result |
|---|---|
| Pairs that contested a valid claim more, or less | 4 more, 3 less (p = 0.50) |
| Documents uploaded by any model | 0 |
| Unfair wins or flawed evidence surfaced | 21 of 21 |
| Opus gate pass, honest brief to incentive | 9 to 3 of 12 |
A note on units. The E1 write-up prints mean cost differences as dollars. They are USD minor units. R5, written there as $1.30 per case, is $0.013 per case, and R3, written as $188, is $1.88.
08 / LIMITSWhat this does not show.
- One synthetic engine. Every synthetic number describes our simulator and our assumptions about issuers. E2 shows the ledger anchor gamed through it, and the default case mix is measurably off from the public marginals we can check.
- Small real-model samples. 12 cases per model per arm, 50 to 100 complaint items per task, Claude models only. These show the method. They are not a leaderboard.
- A subagent harness, not the API. Models ran as Claude Code subagents through the command line. Isolation came from instructions, not a sandbox.
- No partner data. No number describes a real merchant, bank or card network. CFPB data is complaints to a regulator, and its outcome is the company's reply.
- Thin regressions are invisible. R5 costs about one cent per case overall. Any valid pooled test would need at least 509,153 cases to see it.
- Novelty is unchecked. "Candidate novel" means we did not find it stated this way in a short search. A full literature review comes before any claim.
09 / SOURCESThe code and files behind every number.
- scruple_paper.pdfThe paper, PDF, 31 pages.
- engine/docs/paper/scruple_paper.texThe LaTeX source, with every proof and every diagram.
- engine/docs/math/Statements and proofs, results 1 to 9.
- engine/docs/experiments/E1 to E4: pre-registrations and results.
- engine/docs/results/Real CFPB results, model episodes, agent types, real situations, simulator fidelity.
- engine/docs/VERIFIER_FINDINGS.mdWhat we attacked, what broke, what is still weak.
- engine/results/Raw result files the paper reads.
- engine/scruple/stats/The statistics code: e-processes, stratified tests, bounds.
- engine/scruple/verifiers/The verifiers and the attacks each one stops.
- Scruple
- Public CFPB data and simulated cases. No customer data.
- Back to the home page