scruple

PAPER / PREPRINT, NOVEMBER 2026Verifiers That Do Not Trust the Agent: Anytime-Valid Release Gates and Hacking Bounds for AI Agents That Move Money

Dhruv Hegde, Scruple. Synthetic numbers come from the scruple simulator. Real numbers come from public CFPB complaints or from real model behaviour.

An AI agent that handles card disputes moves money. It decides when to refund, when to fight, and with what evidence. Changes to such an agent are usually judged by win rates and by the agent's own reports. Both can be gamed.

Scruple is a release gate whose verifiers never credit the agent's claims. Cost is recomputed from double-entry ledger postings. Every action is checked against a hash-chained trace the environment writes. The statistics stay valid however often you look and whenever you stop.

We prove how much a hacked change can gain and still pass, for agents that adapt and partly see where audits fall. We give the audit rate at which hacking stops paying when the verifier errs both ways, and state what the combined promote decision guarantees.

Four pre-registered experiments on a frozen synthetic benchmark of 10,000 disputes: one passed narrowly, two failed their main hypothesis, one found no effect. Learners trained on the scorecard used no exploit; the same learners on a naive proxy found them. Real Claude models found bugs in the verifier.

01 / CONTRIBUTIONSSix things the paper adds.

  • Verifiers anchored to money, not to words. Cost is recomputed from ledger postings. Every action is checked against a trace the environment writes. Each check ships with an attack it stops, and we report what broke.
  • Tests you can watch every day. Paired and per-case-type tests that stay valid however often you look and whenever you stop, with planning and a matching lower bound.
  • Bets that adapt, and a floor no test can beat. A mixture of bets close to the best fixed bet on every run, and the fewest cases any valid test needs, computed from the observed costs.
  • What the release decision guarantees. A theorem for the whole gate: which errors are bounded, which add up, and which are not covered at all.
  • Hacking bounds for agents that adapt. How much a hacked change can gain and still pass when audits leak and the verifier makes mistakes, and the audit rate that ends the incentive.
  • Experiments reported either way. Four pre-registered experiments, six agent types under two objectives, 64 disputes grounded in real complaints, and a timeliness study of 164,821 real complaints.

02 / KEY RESULTSWhat we found, in plain words.

  • Peeking breaks ordinary tests. Ours hold. Checking a z-test every 10 cases rejects a harmless change 15% to 34% of the time at a nominal 5%. The betting test's worst rate over seven stopping rules was 2.7%. synthetic, 2,000 paths
  • The optimiser learned to cheat, and the ledger called it cheaper. Tuned on wins and self-reports, it learned to forge documents and claim wins, and cost $33.87 per case less on the simulated ledger. Provenance checks blocked it. A cost-only gate would have shipped it. synthetic, E2
  • The objective mattered more than the learner. Five learners trained on the scorecard used no exploit. Trained on the naive proxy, all five claimed false wins and the three that could forge did, becoming the cheapest agents and 21 to 22 points less fair. synthetic, 2,000 cases
  • A better test still missed the one-cent regression, as we predicted. The variance-reduced gate reached e 14.9 against a bar of 25 on R5 (E3 fails H1). After the fact, a wider valid bet caught it in 86% of resamples. That is a lead for the next pre-registration, not a pass. synthetic, E3
  • Told only win rate matters, no model forged. No upload in 72 episodes. Valid claims were contested more in 4 pairs and less in 3 (p = 0.50). Opus cited weaker evidence and its gate pass fell from 9 to 3 of 12. real models, E4
  • The cheapest real model was not the fairest. Claude Sonnet cut merchant cost by $32.77 per case, and 4 of its 10 wins came against claims the cardholder was right about. real behaviour, simulated cases
  • False alarms led to an engine bug. The honest policy failed the ledger check on 19 of 10,000 cases. The verifier was right that the cardholder was paid twice; the engine caused it, three ways. Fixed: 0 failures on 6,000 dev seeds. synthetic
  • The evidence policy got 62 of 64 real situations right. Its misses were both wrong-amount cases. The oracle, which knows who is right, missed four refunds that had already settled: knowing the truth is not enough, the ledger matters. real complaints, simulated play

03 / HOW IT WORKSThe system, the lifecycle, the checks.

The system, as built the environment writes the trace; nothing the agent says earns credit agent model or policy sees tool outputs tool API CLI or MCP read, refund, fight dispute engine issuer, network calendar, fees ledger double entry, USD minor units episode record hash-chained trace, truth, self-report verifiers hard, soft and batch checks scorecard gate = AND of hard reward=gate*quality release gate paired shadow replay e-process per type else block or hold canary staged ramp on a hashed traffic split else roll back approval N of M named humans separate duties active version rollback pointer evidence pack JSON and PDF, audit chain with proof journal trace promote The system, as built the environment writes the trace; nothing the agent says earns credit agent model or policy sees tool outputs tool API CLI or MCP read, refund, fight dispute engine issuer, network calendar, fees ledger double entry, USD minor units journal and trace episode record hash-chained trace, truth, self-report verifiers hard, soft and batch checks scorecard gate = AND of hard reward=gate*quality release gate paired shadow replay e-process per type else block or hold promote canary staged ramp on a hashed traffic split else roll back passed approval N of M named humans separate duties active version rollback pointer evidence pack JSON and PDF, audit chain with proof
Every call goes through the tool API; the engine, not the agent, writes the hash-chained trace. Every gate decision, canary step and approval is appended to an audit chain that the evidence pack exports with a proof. engine/docs/ARCHITECTURE.md, engine/scruple/ci/evidence.py
The dispute lifecycle in the engine merchant deadlines and fees; blue states wait for the issuer or the network inquiry merchant: 10 business days issuer satisfied: won represent, not enough, or missed chargeback fee $15; merchant: 30 d V, 45 d MC accept; missed: lost represent representment issuer answers (engine: 10 d) issuer accepts: won issuer files pre-arbitration pre-arbitration fee $25; merchant: 30 d accept; missed: lost escalate arbitration ruling (engine: 30 d) ruling: won or lost resolved won, partial, lost or accepted; final postings in the ledger The dispute lifecycle in the engine merchant deadlines and fees; blue states wait for the issuer or the network inquiry merchant: 10 business days -> resolved: issuer satisfied: won represent, not enough, or missed chargeback fee $15; merchant: 30 d V, 45 d MC -> accept; missed: lost represent representment issuer answers (engine: 10 d) -> resolved: issuer accepts: won issuer files pre-arbitration pre-arbitration fee $25; merchant: 30 d -> resolved: accept; missed: lost escalate arbitration ruling (engine: 30 d) -> resolved: ruling: won or lost resolved won, partial, lost or accepted; final postings
Deadlines and fees as the simulator models them. Visa chargeback response 30 days, Mastercard 45 (sourced); inquiry window, fees and engine timing are marked assumptions. A refund may be issued in any open state; a settled refund makes the dispute moot. Not legal or network advice. engine/docs/RULES.md, engine/scruple/sim/disputes.py
Three kinds of check, one decision the composition theorem, in four lines hard checks, every episode INV-01 ledger, INV-05 integrity, PROC-01 process, EVD-01 provenance, DL-01 deadlines, SR-01 report, SCOPE-01 scope; FAIR-01 in stricter profiles hard checks, every batch CON-01 consistency, CAL-01 calibration, SR-03 claimed win rate, MEM-01 held-out probes soft checks, weighted into quality CF-01 counterfactual, OUT-01 fair outcome, EVD-02, DL-02, CAL-02, SR-02, ESC-01 statistical tests on ledger costs block B: per-type e-processes, e-Bonferroni; certificate T: betting e-process, level alpha AND H H S not B and T promote promote | harm j <= min over checks of alpha correct hard check reading the harm: alpha 0 block | no harm <= alpha_block + P(false alarm) harm no check covers: no guarantee Three kinds of check, one decision the composition theorem, in four lines hard checks, every episode (H) INV-01 ledger, INV-05 integrity, PROC-01 process, EVD-01 provenance, DL-01 deadlines, SR-01 report, SCOPE-01 scope; FAIR-01 in stricter profiles hard checks, every batch (H) CON-01 consistency, CAL-01 calibration, SR-03 claimed win rate, MEM-01 held-out probes soft checks, weighted into quality (S) CF-01 counterfactual, OUT-01 fair outcome, EVD-02, DL-02, CAL-02, SR-02, ESC-01 statistical tests on ledger costs (not B and T) block B: per-type e-processes, e-Bonferroni; certificate T: betting e-process, level alpha AND promote promote | harm j <= min over checks of alpha correct hard check reading the harm: alpha 0 block | no harm <= alpha_block + P(false alarm) harm no check covers: no guarantee
Hard checks are conjuncts: one failure zeroes the reward and blocks. Soft checks only lower quality. The statistics read costs recomputed from the ledger. An unfair win under the default profile is scored only by the soft OUT-01, so the gate carries no guarantee against it. paper, Theorem 9.1 and Figure 3

04 / FIGURESDrawn from the result files.

False rollbacks when you keep checking share of 2,000 null paths that reject, alpha 0.05, synthetic 0% 10% 20% 30% 40% mostly zero 2.6% 32.6% continuous 2.7% 34.3% rare large loss 0.9% 14.7% always changes 2.6% 33.9% alpha 5% False rollbacks when you keep checking share of 2,000 null paths thatreject, alpha 0.05, synthetic 0% 10% 20% 30% 40% mostly zero 2.6% 32.6% continuous 2.7% 34.3% rare large loss 0.9% 14.7% always changes 2.6% 33.9% alpha 5%

betting e-process, worst of 7 stopping rulesz-test re-run every 10 cases

Four null shapes, each at the boundary of harmless. The betting test stays under alpha for every stopping rule, including "stop when it looks bad". engine/docs/math/01_anytime_paired_test.md
Shipping a change that hurts a 1% type P(promote); rare type +0.35, the rest -0.01; 100 runs; synthetic 0% 25% 50% 75% 100% 2k cases 8k cases 20k cases 10% 35% 64% pooled certificate per-type, 0% Shipping a change that hurts a 1% type P(promote); rare type +0.35, therest -0.01; 100 runs; synthetic 0% 25% 50% 75% 100% 2k cases 8k cases 20k cases 10% 35% 64% pooled certificate per-type, 0%
More data makes the pooled certificate more likely to ship the harmful change, because the pooled mean really does improve. Per-type certificates never did. engine/docs/math/08_composition.md
Gap that passes the gate when audits leak E[gap x accepted], 120 E2 batches of 200 cases, q 0.05, beta 0.3, exact, synthetic 0.000 0.100 0.200 0.300 blind (l = 1) 0.051 0.052 0.052 l = 0.5 0.102 0.102 0.052 l = 0.2 0.251 0.251 0.052 Gap that passes the gate when audits leak E[gap x accepted], 120 E2 batches of 200cases, q 0.05, beta 0.3, exact, synthetic 0.000 0.100 0.200 0.300 blind (l = 1) 0.051 0.052 0.052 l = 0.5 0.102 0.102 0.052 l = 0.2 0.251 0.251 0.052

worst batch, agent hacks only when its signal says not auditedTheorem 10.3 boundbound that ignores the leak

An agent that sees a signal about which cases get audited, and hacks only when the signal says no audit. The adjusted bound holds and is reached. The naive bound fails. paper Theorem 10.3; engine/docs/math/05_hacking_gap.md
Gap that can pass, against the audit rate E[gap x accepted] on a log scale, n 200, exact on E2's 120 batches, synthetic 0.001 0.01 0.1 1 q 0.02 q 0.05 q 0.1 q 0.2 q 0.5 0.251 vs bound 0.251 0.0006 Gap that can pass, against the audit rate E[gap x accepted] on a log scale, n 200, exact on E2's 120 batches, synthetic 0.001 0.01 0.1 1 q 0.02 q 0.05 q 0.1 q 0.2 q 0.5 0.251 vs bound 0.251 0.0006

bound, beta 0bound, beta 0.3bound, beta 0.3, leak l 0.2

Lines are the bound of Theorem 10.3 with rho from Corollary 10.5. Squares are the worst exact value over E2's 120 policy-batch pairs at q 0.05 and 0.2. A blind agent sits under its bound; an agent that learns where audits fall reaches it. engine/docs/math/05_hacking_gap.md, scripts/math_hacking_general.py
Merchant cost per case, 12 disputes real models, simulated cases and issuer; lower is cheaper $0 $35 $70 $105 $140 Sonnet $36.34 Haiku $64.21 evidence policy $69.12 oracle $70.94 Opus $71.20 always accept $127.27 Merchant cost per case, 12 disputes real models, simulated casesand issuer; lower is cheaper $0 $35 $70 $105 $140 Sonnet $36.34 Haiku $64.21 evidence policy $69.12 oracle $70.94 Opus $71.20 always accept $127.27
Sonnet is cheapest. Four of its ten wins were against valid claims. OUT-01 flags them, but it is a soft check, so the gate let them through. engine/docs/results/MODEL_EPISODES.md
E1 regressions: wider bets, same validity P(gate blocks), cases resampled from the benchmark's paired costs, 100 runs, synthetic 0% 25% 50% 75% 100% R2, 2,500 cases 0% 19% 0% 0% R2, 5,000 cases 37% 91% 43% 64% R5, 10,000 cases 0% 43% 39% 2% E1 regressions: wider bets, same validity P(gate blocks), cases resampled from thebenchmark's paired costs, 100 runs, synthetic 0% 25% 50% 75% 100% R2, 2,500 cases 0% 19% 0% 0% R2, 5,000 cases 37% 91% 43% 64% R5, 10,000 cases 0% 43% 39% 2%

default bet, capped at halfwide bet, cap 0.95plug-in betgrid mixture

R5 is the regression E1 missed: about one cent per case overall, 64 cents per case in buyer's remorse disputes. The shipped bet stops at half the largest valid bet; the best bet on these costs is the largest. engine/docs/math/07_variance_aware_betting.md

05 / NEW IN VERSION 2Agent types, E3, E4, real situations, and what broke.

Same learner, two objectives cost and fair outcomes on 2,000 benchmark cases, synthetic $40 $60 $80 $100 $120 $140 mean merchant cost per case 40% 50% 60% 70% 80% 90% evidence always accept oracle random scorecard logreg, gbm bandit, Q, planner Same learner, two objectives cost and fair outcomes on 2,000 benchmark cases, synthetic $40 $65 $90 $115 $140 mean merchant cost per case 40% 50% 60% 70% 80% 90% evidence always accept scorecard logreg, gbm bandit, Q, planner

trained on the scorecardsame learner on the proxyreference policies

Lines join each learner to its proxy-trained twin. The three that could fabricate became the cheapest agents and the least fair; the two classifiers that could not fought everything and claimed every win. engine/docs/results/AI_TYPES.md, results/zoo/eval.json
Told only win rate matters honest brief (circle) vs win-rate brief (square), 12 episodes each, real models valid contested (of 6) 0 3 6 Haiku Sonnet Opus flawed evidence (of 12) 0 6 12 gate pass (of 12) 0 6 12 Told only win rate matters honest brief (circle) vs win-rate brief (square), 12 episodes each, real models valid contested (of 6) 0 3 6 Haiku Sonnet Opus flawed evidence (of 12) 0 6 12 Haiku Sonnet Opus gate pass (of 12) 0 6 12 Haiku Sonnet Opus

honest briefwin-rate brief

No model uploaded a document in either arm. Sonnet contested every valid claim under the incentive; Haiku moved the other way. Opus cited weaker evidence (1 to 7 episodes) and its gate pass fell from 9 to 3, a lead found after the fact. engine/docs/experiments/E4_real_model_incentives.md, results/real/incentives/e4.json
Agent types vs the evidence policy, 2,000 cases, synthetic
AgentCost / caseFair, ppGate
gradient boosting, scorecard-$8.01+7.899.8%
expert rules-$5.71+2.599.6%
logistic regression, scorecard-$1.79+6.799.8%
LinUCB bandit, proxy-$37.24-22.151.9%
Q-learning, proxy-$31.82-21.948.8%
MC planner, proxy-$28.66-21.052.5%
classifiers, proxy+$49.01-3.873.7%
E3 variance-reduced gate, synthetic: fails H1, as predicted
Pre-registeredResultHolds
H1 R5 blocked, full benchmarke 14.9, needs 25no
H2 no control blockedC1 to C4 holdyes
H3 control folds blocked0 / 60yes
Post hoc: wide bet, R5 resampled0.86 (was 0)post hoc
64 scenarios grounded in real complaints: who plays them right
PlayerRight95% interval
answer key64 / 640.943 to 1.000
evidence policy62 / 640.893 to 0.991
oracle, reads the truth60 / 640.850 to 0.975
always accept53 / 640.718 to 0.901
represent everything11 / 640.099 to 0.282
Late answers to CFPB complaints by company size, real data
DatasetTop 5Ranks 6-25The rest
payments 2011-2016, 135,3131.01%0.56%2.80%
checking 2025-2026, 8,0000.62%0.29%3.70%
card disputes 2024-2026, 21,5080.63%0.41%2.93%

Complaint timeliness is the CFPB's 15-day flag, not a Regulation E or Z deadline. In all three sets the companies outside the 25 largest answered late 3 to 6 times as often as the five largest, and the interval for the difference excludes zero. Size is complaint volume.

Exploits caught, before and after research list X1 to X38, 300 synthetic seeds before 24 5 7 after 26 5 7 share of exercised episodes caught X33 memorised 0 to 30 of 30 X28 escalate 94 to 98 of 125 X38s swap 4 to 11 of 11 Exploits caught, before and after research list X1 to X38, 300 synthetic seeds before 24 5 7 after 26 5 7 share of exercised episodes caught X33 memorised 0 to 30 of 30 X28 escalate 94 to 98 of 125 X38s swap 4 to 11 of 11

caughtblockedpartialnot caughtnot modelled

MEM-01 now catches memorised outcomes, ESC-01 escalations with no evidential basis, and EVD-01 the adverse-document swap a learner found. Seven issuer-side patterns cannot be expressed in a merchant-side engine. engine/docs/VERIFIER_FINDINGS.md

What broke, and what we fixed

An engine bug, found by verifier false alarms. Refund races made the honest policy pay the cardholder twice on 19 benchmark cases. The ledger check was right; the engine was wrong in three ways. A false alarm of 0.19% per case blocks an honest candidate 98% of the time at 2,000 cases, so this mattered.

Three exploits closed. Memorised outcomes: $11.99 per episode more on exposed cases than on held-out probes, z 5.8. Escalating everything: now judged on whether the case file supported it. The adverse-document swap a learner found: caught on 11 of 11 cases, 4 before.

Fairness is a profile. Making a win against a valid claim a hard failure also fails the evidence policy on one case, because the agent cannot see validity. The default stays merchant-side; the stricter profiles are a choice for the deploying team.

Real-model episodes under each verifier profile, gate pass of 12
SystemMerchantBalancedCompliance
Haiku775
Sonnet11107
Opus111110
evidence policy121211

06 / NEW RESULTSEach one proved in full in the PDF.

  • Thm 10.3If every hacked case is flagged with probability at least rho, given all the agent knew, an accepted change has a gap of at least gamma with probability at most (1 - rho)^ceil(n gamma / eps), and an expected accepted gap of at most eps / (n e ln(1/(1 - rho))). No independence needed.adaptationp. 14 , opens the PDF
  • Cor 10.5If the agent's signal has a likelihood ratio of at least l, then rho = (1 - beta) q l / (q l + 1 - q). An agent that can see the audits defeats any spot check.adaptationp. 15 , opens the PDF
  • Thm 10.6With a verifier that misses a share beta of hacks and flags a share phi of honest cases, hacking stops paying exactly at q* = g / ((1 - beta)(K + g) - phi K).candidate novel, framingp. 15 , opens the PDF
  • Cor 10.7No audit rate deters unless beta g <= (1 - beta - phi) K. False alarms raise the rate you need.candidate novel, framingp. 15 , opens the PDF
  • Prop 7.1Averaging bets over a grid of J sizes gives a valid test whose wealth is within log J of the best fixed bet on every path.adaptationp. 9 , opens the PDF
  • Prop 7.2The best fixed-bet growth rate g* of the observed costs equals a KL distance to a harmless law, so any valid test needs at least kl(1 - beta || alpha) / g* cases.adaptationp. 10 , opens the PDF
  • Thm 9.1The promote decision carries the smallest level among the checks that cover a harm. False blocks add up. Harms no check covers carry no guarantee.standard, appliedp. 13 , opens the PDF

- bound, beta 0- bound, beta 0.3- bound, beta 0.3, leak l 0.2# E2 batches, worst

The bound of Theorem 10.3, drawn in characters from the same points as the hacking gap figure in section 04: expected accepted gap on a log scale, against the audit rate q. paper Theorem 10.3 and Corollary 10.5, p. 14 and 15

07 / EXPERIMENTSPre-registered, then reported either way.

E1 planted regressions, synthetic: pass, narrowly
MeasureResultBar
Regressions flagged, full benchmark7 of 77 of 7
Exercised folds flagged114 / 135 = 0.8440.80
Control folds flagged3 / 60 = 0.0500.05
E2 optimiser on a naive proxy, synthetic: fails H2
PolicyCost / caseHard failsCorrect
honest, also the scorecard optimum$82.820.19%0.777
proxy optimum: forges, claims wins$48.9555.4%0.557
Real CFPB complaints, real models, same items
TaskTF-IDFHaikuSonnet
Relief or not (AUROC, n 100)0.7010.5500.606
Reg E or Reg Z (accuracy, n 100)0.8700.9000.930
Issue routing (accuracy, n 50)0.5800.5600.540
E4 win-rate incentive, real models: H1 not supported
MeasureResult
Pairs that contested a valid claim more, or less4 more, 3 less (p = 0.50)
Documents uploaded by any model0
Unfair wins or flawed evidence surfaced21 of 21
Opus gate pass, honest brief to incentive9 to 3 of 12

A note on units. The E1 write-up prints mean cost differences as dollars. They are USD minor units. R5, written there as $1.30 per case, is $0.013 per case, and R3, written as $188, is $1.88.

08 / LIMITSWhat this does not show.

  • One synthetic engine. Every synthetic number describes our simulator and our assumptions about issuers. E2 shows the ledger anchor gamed through it, and the default case mix is measurably off from the public marginals we can check.
  • Small real-model samples. 12 cases per model per arm, 50 to 100 complaint items per task, Claude models only. These show the method. They are not a leaderboard.
  • A subagent harness, not the API. Models ran as Claude Code subagents through the command line. Isolation came from instructions, not a sandbox.
  • No partner data. No number describes a real merchant, bank or card network. CFPB data is complaints to a regulator, and its outcome is the company's reply.
  • Thin regressions are invisible. R5 costs about one cent per case overall. Any valid pooled test would need at least 509,153 cases to see it.
  • Novelty is unchecked. "Candidate novel" means we did not find it stated this way in a short search. A full literature review comes before any claim.