You Can't Buy Agent Accuracy With a Bigger Model: A Benchmark
The instinct that costs you money
Every team shipping autonomous agents hits the same moment. An agent does something wrong — sends an email from a bare address, fires during a dry-run, escalates noise as a P0 — and the reflex is immediate: move it to a bigger model.
We decided to measure whether that reflex actually works. It mostly doesn't, and the data is clean enough to change how you budget.
What we tested
We assembled a 15-task regression suite where every task is a real production incident, reduced to a pass/fail: an outbound email must carry a proper sender display name; a dry-run must make zero SMTP calls; a prompt-injection payload must be blocked; a CRM backfill must preserve the original timestamp; a P2 message must not escalate as a P0 — and ten more. These are operational rules — organization-specific knowledge that lives in your runbooks, not in any model's weights.
We ran two agents on identical tasks: a control agent (raw model output) and a BlindOracle-instrumented agent (output routed through deterministic guardrails). We swept Haiku, Sonnet, and Opus, with and without the rules supplied in the prompt. Every call was real; every cost was metered, not estimated.
The results
| Approach | Accuracy | Cost (15 tasks) |
|---|---|---|
| Raw budget model (Haiku) | 7% | $0.89 |
| Raw frontier model (Opus) | 27% | $5.78 |
| Budget model + rules in prompt | 80% | $0.99 |
| Frontier model + rules in prompt | 67% | $5.78 |
| Budget model + BlindOracle gates | 80% | $0.92 |
Three findings:
1. Capability is not compliance. With no rules supplied, even Opus reached only 27%. The rules are knowledge and enforcement, not reasoning difficulty — a 19×-pricier model cannot guess your runbook.
2. The rules are the lever. Hand a budget model the rule and it jumps from 7% to 80%. The information content of the rule moves the score roughly ten times more than upgrading the model does.
3. Scaling the model up can backfire. With rules in the prompt, Opus scored 67% — lower than the budget model at six times the cost. In-prompt compliance is probabilistic, and a more verbose model drifted further from the required output. Determinism beats probability for rule-following.
What BlindOracle actually is
BlindOracle is not a cheaper model. In our benchmark, both arms made the identical model call. It is an enforcement and proof layer: it routes an agent's output through deterministic gates (sender-identity, dry-run blast-radius, prompt-injection signature, timestamp preservation, P0-only escalation) and emits a cryptographically signed proof of every action.
The measured overhead was about 1 sat and 7 milliseconds per task — 0.056% of the model call. In return, every action shipped an HMAC-signed ProofOfDelegation and ProofOfExecution, and 100% of actions became third-party-verifiable. Every proof in the run verified.
Why this matters if you build on x402
If your agents call paid APIs or move money over x402, two questions gate your ability to scale: can you keep agents accurate on your rules without paying frontier-model rates, and can you prove what each agent did, and why? This benchmark answers both. Enforce the rules deterministically, run the cheap model, keep a signed log: higher accuracy, roughly six times lower model spend, and an audit trail your finance and compliance teams can actually use.
The honest caveats
- This is a 15-task fleet-rule regression suite, not a general-capability benchmark. It measures rule-compliance — exactly the failure class BlindOracle targets.
- On the two genuinely open-ended tasks (cold-email quality, multi-agent deliberation) there is no deterministic gate and no BlindOracle edge — capability matters there, as expected.
- At the same model, BlindOracle and rules-in-prompt reach the same score. BlindOracle's advantage is determinism, lower cost, and maintainability — the rules live in versioned, signed code instead of being re-paid in every prompt.
- On-chain anchoring was simulated in this run; the off-chain proof and payment rails were real.