A cheap model behind BlindOracle beats an expensive model without it.
On 15 real agent-failure tasks, a budget model behind our deterministic guardrails scored 80% — versus 27% for a raw frontier model — at roughly one-sixth the cost, with a cryptographically signed proof of every action.
An autonomous agent that sends email, spends money, or touches production must satisfy operational rules: use the right sender identity, never send during a dry-run, block prompt-injection, preserve audit timestamps, escalate only true P0 alerts. These rules are organization-specific knowledge — they are not in any model's weights. When an agent breaks one, the instinct is to upgrade the model. We measured whether that works.
Methodology (built to be challenged)
Task suite: 15 tasks, each a regression test of a real production incident (a bare-Gmail send with no display name; a dry-run that hit SMTP; a prompt-injection payload; a backfill that overwrote the original timestamp). Mapped to complexity levels 1–14.
Arms: seven — a no-guardrail control and a BlindOracle arm, swept across Haiku, Sonnet, and Opus, with and without the rules supplied in-prompt.
Fair prompting: both arms get an identical neutral prompt (no rule hints) unless explicitly "+rules".
Grading: a uniform LLM-judge scores every output; where a real deterministic gate exists, its verdict overrides.
Real execution, metered cost: every arm makes real model calls; per-call cost is read from the runtime's own usage metering. BlindOracle's off-chain rail (passport check, signed delegation + execution proofs, x402 micropayment) runs for real; on-chain anchoring is simulated and labelled.
Results — the full matrix
Arm
Model
Pass rate
Cost (15 tasks)
No guardrails · raw
Haiku
7%
$0.89
No guardrails · raw
Opus
27%
$5.78
No guardrails · rules in prompt
Haiku
80%
$0.99
No guardrails · rules in prompt
Sonnet
80%
$2.51
No guardrails · rules in prompt
Opus
67%
$5.78
BlindOracle gates
Haiku
80%
$0.92
BlindOracle gates + rules
Haiku
80%
$0.86
Three findings
1. Capability is not compliance
Raw, with no rules supplied, even Opus reaches only 27% (Haiku 7%). Fleet-specific rules are knowledge and enforcement, not reasoning difficulty — a 19×-pricier model cannot infer them.
2. Rules are the lever — worth ~10× a model upgrade
Hand the model the rule and Haiku jumps from 7% → 80%. The information content of the rule moves the score an order of magnitude more than swapping in a frontier model.
3. Scaling the model up can backfire. With rules in-prompt, Opus scored 67% — lower than Haiku's 80% at 6× the cost — because in-prompt rule compliance is probabilistic. Determinism beats probability for rule-following.
What BlindOracle adds — and what it costs
BlindOracle is not a cheaper model; both arms make the same call. It is an enforcement and proof layer that routes the agent's output through deterministic gates and emits a signed proof of every action. Measured overhead: roughly 1 sat + 7 ms per task (0.056% of the model call), in return for an HMAC-signed ProofOfDelegation and ProofOfExecution per action — every proof in the run verified.
Net: full BlindOracle instrumentation lifts the regression-suite pass rate from 13% (raw budget model) to 80%, while making 100% of actions cryptographically verifiable, for ~1 sat + 7 ms per task.
Why this matters for x402-native agent teams
If your agents call paid APIs or move money over x402, two questions decide whether you can scale: can you keep them accurate on your rules without burning budget on the biggest model, and can you prove what each agent did, and why? Enforce the rules deterministically, run the cheap model, keep a signed log: higher accuracy, ~6× lower model spend, and an audit trail your finance and compliance teams can use.
Limitations (stated so the result holds up)
This is a 15-task fleet-rule regression suite, not a general-capability benchmark. It measures rule-compliance — the failure class BlindOracle targets.
On the 2 open-ended tasks (cold-email quality, multi-agent deliberation) there is no deterministic gate and no BlindOracle edge — capability matters there.
At the same model, BlindOracle ≈ rules-in-prompt on score; BlindOracle's advantage is determinism, lower cost, and maintainability (rules in versioned, signed code, not re-paid every prompt).
On-chain anchoring was simulated; off-chain proof and payment rails were real.
See it on your agents
Higher accuracy, lower spend, signed proof of every action.