Ox Alpha Benchmarks: What the Numbers Actually Say
The viral '80% on DeepSWE — beats GPT-5.6 Sol' claim came from a 10-task sample. Full 113-task runs measured 58.4%. Here's every number, the tool-call failure caveat, and what it means for agent work.
Short answer: ox-alpha did not beat GPT-5.6 Sol. The viral 80% came from a 10-task sample its own author flagged as high-variance. Full 113-task runs put ox-alpha at 58.4% (95% CI: 49.2–67.1%), with larger evaluations converging around 62.8–63% — DeepSeek-V4-Pro territory, a few steps below the frontier leaders. At $0 per token, that's still the best value in coding models right now. Here's the full picture.
How the 80% headline happened
Days after ox-alpha appeared on OpenRouter, a developer running as @davis7 posted a DeepSWE comparison: ox-alpha above 80%, against Claude Fable 5 at 65% and GPT-5.6-sol at 52%. Screenshots traveled; the "stealth model beats the frontier" take wrote itself.
To @davis7's credit, the post itself warned that a 10-task sample carries huge variance. The full runs proved the warning right.
Connect the Claude or Codex you already pay for — the rest runs on workers that cost a fraction.
Download meshcode →What full runs measured
An independent complete pass over all 113 DeepSWE tasks (MatchaOnMuffins/oxalpha) measured:
- 58.4% resolve rate (66/113), 95% CI [49.2%, 67.1%]
- 80% of tasks landed ≥90% of their fail-to-pass tests — when it solves a task, it usually really solves it
- 9.7% of the benchmark was lost to tool-call format failures — not reasoning failures
Larger third-party evaluations converged around ~62.8–63%. The honest read: ox-alpha sits in the low 60s, with real uncertainty at the edges of that confidence interval.
Where that lands on the leaderboard
| Model (configuration) | DeepSWE |
|---|---|
| Claude Opus 5, max reasoning | 74% ±4% |
| GPT-5.6 Sol, max reasoning | 73% ±3% |
| Claude Fable 5 | 70% ±4% |
| GLM-5.3 / Kimi K3 | 69% |
| DeepSeek V4 Pro | 63% ±6% |
| Ox Alpha (third-party runs) | ~58–63% |
Two caveats belong under that table. First, ox-alpha has no official leaderboard entry — every number is third-party, on someone else's harness. Second, the early "level with GPT-5.6-sol" comparisons were against mid-tier Sol configurations; Sol at max reasoning is still ~10 points clear.
The most interesting number isn't the score
It's the 9.7% lost to tool-call format failures. Those are tasks where the model's reasoning was fine but its tool calls came out malformed and the harness scored them as failures. Two implications:
- The ceiling is higher than the score. Nearly 10 points of headroom is plumbing, not intelligence — and plumbing improves fast, on both sides (model updates and harness fixes).
- Your harness matters. A coding agent that streams real tool calls, validates them, and retries cleanly recovers points a bare API loop drops. This is exactly where a full agentic environment earns its keep — you can watch every tool call land in real time instead of guessing what the model "really" meant.
The verdict for practitioners
- Not "beats GPT-5.6 Sol." Anyone still repeating the 80% line is quoting a 10-task sample.
- Yes to "DeepSeek-V4-Pro-class performance at $0" — which, per point of DeepSWE, makes it the best-priced coding model available today.
- Best suited to volume agent work: long sessions, parallel panes, fan-out subagents — the workloads where cost dominates and a low-60s model with a 1M context window is more than enough.
Want the mystery side? Who made ox alpha collects every attribution clue, ranked. And to actually run it — four panes in parallel, $0 per token — see ox-alpha in MeshCode.
Benchmark figures are third-party measurements as of August 26, 2026, not official leaderboard entries, and may shift as harnesses and model versions change. Ox Alpha is distributed anonymously via OpenRouter; MeshCode is not affiliated with its developer.