Supabase Just Open-Sourced the First Real-World Benchmark for AI Coding Agents

TL;DR

Supabase just open-sourced Supabase Evals — a benchmark framework that tests AI coding agents against real-world backend tasks instead of isolated code completion. Claude Code leads at 53.2% success rate, Codex follows close behind. The framework is Apache-2.0 licensed and anyone can run it locally. For developers evaluating which AI coding agent to trust with production work, this is the closest thing to a real-world test we have.

What Happened

On August 1, Supabase released supabase/evals on GitHub under an Apache-2.0 license. It is a reproducible benchmark harness that pits AI coding agents against actual Supabase development tasks — not toy problems, not synthetic datasets, but tasks derived from real support tickets and GitHub issues.

The agents tested: Claude Code, OpenAI Codex, and OpenCode. The tasks: building a database schema from scratch, debugging a failed Edge Function, fixing a broken Row-Level Security policy, writing migrations, handling auth flows. These are the exact problems developers pay Supabase to solve.

The benchmark runs locally through pnpm. Each agent gets access to the Supabase MCP server and CLI, same tooling a human developer would use. The framework scores whether the agent completed the task correctly, not just whether the code compiles. A web dashboard publishes the results for anyone to inspect.

The Results

Claude Code completed 53.2% of scenarios successfully — the highest score. Codex followed closely behind. OpenCode, the fully open-source alternative, lagged but offers complete transparency on how it operates.

A 53.2% pass rate sounds low at first glance, but the context matters. These are not isolated code-completion prompts. Each task requires the agent to understand a stateful environment, make decisions across multiple steps, and produce working infrastructure — not just code. A human developer would not get 100% on these on first attempt either.

Why This Benchmark Is Different

Most AI coding benchmarks — SWE-bench, LiveCodeBench, Terminal-Bench — test models on isolated code tasks. The agent gets a prompt, produces code, and a judge scores the output. These benchmarks are useful but they miss something fundamental: real development happens in stateful environments where one wrong decision cascades.

Supabase Evals tests the full loop. The agent has to set up a project, understand the current state of a database, figure out what is broken, make changes, and verify the fix works. It is closer to what actually happens when you type "fix the auth middleware" into Claude Code at 10pm on a Thursday.

The framework is also reproducible. Anyone can clone the repo, run the same tests, and get the same results. This matters because vendor-published benchmarks have an obvious incentive problem. An independent, open-source benchmark that anyone can verify changes the dynamic.

What This Means for Developers

If you are evaluating AI coding agents, you now have an additional data point that is qualitatively different from SWE-bench scores. Here is how to use it:

1. Decide whether backend and infrastructure tasks matter for your workflow. If you mostly write frontend components, Supabase Evals results are less relevant than if you are building APIs and database schemas.
2. Do not over-index on the exact numbers. The difference between 53.2% and whatever Codex scored is less important than the fact that no agent is above 60%. The takeaway is "these tools are useful but still need human oversight for complex backend work" — not "Claude Code is definitively better."
3. If you use Supabase, run the benchmark yourself. The framework is open source. Your specific stack and workflows will stress agents differently than the default scenarios.
4. Watch what happens next. Supabase built a benchmark for their platform. If other platforms — Vercel, AWS, Firebase — follow suit, we will get a much richer picture of how AI coding agents handle different infrastructure contexts.

The Bigger Picture

This release is part of a broader shift in how we evaluate AI coding tools. Six months ago, the conversation was dominated by model-versus-model benchmark tables. Now we are seeing platform-specific, task-specific evaluation frameworks that measure what agents can actually build.

Supabase's move is also strategically smart. By open-sourcing the benchmark, they create an incentive for AI coding tool makers to optimize for Supabase compatibility. If Claude Code wants to top the leaderboard, it needs to get better at writing Supabase Edge Functions and RLS policies. That is a moat disguised as a public service.

The benchmark gap between top agents is narrowing. Claude Code at 53.2% and Codex close behind suggests the frontier is converging. The next frontier in AI coding is not better models — it is better evaluation of what those models can actually ship.

Bottom Line

Supabase Evals gives developers a real-world benchmark for AI coding agents on backend tasks. Claude Code leads, no agent exceeds 60%, and the framework is open source. Run it yourself if Supabase is part of your stack. If not, watch for similar platform-specific benchmarks — they are the next evolution of AI coding tool evaluation.