Supabase Evals: An AI Code Assistant Benchmark for Developers

Supabase has officially open-sourced Supabase Evals, a groundbreaking benchmark and framework designed to rigorously test and score the performance of AI code assistants like Claude Code, Codex, and OpenCode against authentic Supabase development challenges, providing invaluable data for Software Developers seeking to optimize their workflows.

• Supabase Evals is now publicly available under the Apache-2.0 license, allowing any Software Developer to run it locally and contribute.
• The benchmark differentiates itself by testing AI agents against real, containerized Supabase environments rather than simulated mocks, ensuring practical relevance.
• Scoring combines deterministic checks with an LLM-as-a-judge approach, allowing for nuanced evaluation of AI-generated code.
• While top-tier AI models showed strong baseline performance, the integration of ‘skills’ significantly enhanced the capabilities of smaller models, bridging performance gaps.

Introducing Supabase Evals: A New Benchmark for AI Code Assistants

In a significant move for the developer community, Supabase has unveiled Supabase Evals, an open-source initiative aimed at providing a transparent and practical evaluation framework for AI code assistants. This new benchmark directly addresses a growing need for reliable metrics in the rapidly evolving landscape of AI tools for developers. By simulating real-world scenarios, Supabase Evals offers a unique perspective on how effectively AI agents can assist with common development tasks within the Supabase ecosystem, from database schema design to debugging critical edge functions. This initiative is particularly relevant for Software Developers who rely on AI for coding assistance and need to understand the true capabilities and limitations of these sophisticated tools.

The framework is not merely a theoretical exercise; it powers a public leaderboard on supabase.com/evals and serves as an internal regression suite, monitored daily by Supabase. Its deployability is immediate, with the supabase/evals repository accessible under an Apache-2.0 license, allowing any Software Developer to run it locally using pnpm. This accessibility makes it a valuable resource for individual developers, teams, and even organizations in regulated industries like fintech or healthcare, where the accuracy of AI-generated code, especially concerning security policies like Row Level Security (RLS), is paramount.

How Supabase Evals Tests AI Agents on Real-World Scenarios

Supabase Evals employs a sophisticated methodology to assess AI code generation capabilities. It defines three core dimensions: Supabase products (e.g., database, auth, storage, edge functions), development topics (e.g., RLS, security, SQL, SDKs), and development stages (e.g., build, deploy, investigate, resolve). By creating a minimum set of scenarios that touch each of these dimensions, grounded in actual support tickets, bug reports, and GitHub issues, the benchmark ensures its relevance to everyday Software Developer challenges.

These scenarios are categorized into two suites: ‘Benchmark’ scenarios, which cover a broad range of tasks and are publicly published, and ‘Regression’ scenarios, which target known failure modes, refresh daily, and do not influence public scores. Crucially, every scenario runs within a genuine Supabase environment. The framework spins up a hosted-like stack and a local CLI project in containers, allowing AI agents to interact with the actual Supabase Management API-compatible surface. Scoring is a hybrid process, combining deterministic checks for objective correctness with an LLM-as-a-judge model for more nuanced evaluations, granting agents one retry before final grading. This robust setup provides a realistic testing ground for various coding AI solutions.

Initial Performance Insights and Identified Weaknesses in AI Code Generation

The initial findings from Supabase Evals offer valuable insights into the current state of AI code assistants. Many agents demonstrated strong baseline performance, successfully completing a majority of scenarios even without specific ‘skills’ loaded. Notably, top models like Opus 5 and Kimi K3 achieved a 100% score in the ‘Build’ stage unaided. For other models, the integration of ‘skills’ proved highly effective in closing performance gaps; for instance, Sonnet 5 improved from 78% to 100%, and GPT-5.6 Sol saw an increase from 89% to 100%. This highlights the potential of tailored guidance and context for enhancing coding AI.

However, the evaluations also exposed several common weaknesses across the tested AI agents. A recurring issue was the tendency for agents to manually write database migrations instead of leveraging declarative schema approaches, prompting Supabase to recommend skill guidance updates. Similarly, agents often verified authentication manually instead of utilizing the @supabase/server package, suggesting a need for better package selection guidance. Furthermore, there was a noticeable disparity in documentation usage: models like Codex and GPT-5.6 accessed approximately eight documentation pages per scenario, whereas Claude Code consulted fewer than four, checking docs in under 40% of scenarios even with skills enabled. This indicates varying strategies in how different AI tools for developers approach problem-solving and information retrieval.

Practical Implications for Software Developers and the Future of AI Tools

For Software Developers, Supabase Evals presents a vital new resource for navigating the complex world of AI code generation. The practical takeaway is clear: this benchmark provides a standardized, real-world testing ground to compare and contrast various AI code assistant options. Whether you are evaluating GitHub Copilot alternatives like Cursor, Tabnine, or Amazon CodeWhisperer, or simply looking to understand the current capabilities of tools like Claude Code and OpenCode, Supabase Evals offers data-driven insights. It can inform decisions on which AI tools to integrate into development workflows, highlight areas where AI assistance is strong, and identify where human oversight or specialized skills are still indispensable.

The open-source nature of Supabase Evals also empowers the developer community to contribute to its evolution, helping to refine scenarios and expand its coverage. This collaborative approach ensures that the benchmark remains relevant and reflective of emerging challenges in cloud infrastructure and data platforms. As AI debugging tools and coding AI continue to advance, frameworks like Supabase Evals will be crucial in ensuring these technologies genuinely enhance developer productivity and security, rather than introducing new vulnerabilities or inefficiencies. Software Developers can leverage this tool to make informed choices, push the boundaries of AI-assisted development, and shape the future of coding.

Frequently Asked Questions

How can Supabase Evals help me choose the best AI code assistant for my projects?

Supabase Evals provides a public leaderboard and a framework for testing AI agents against real Supabase tasks, allowing you to compare performance metrics of various AI code assistants like Claude Code, Codex, and OpenCode on practical development scenarios.

What kind of development tasks does Supabase Evals evaluate AI agents on?

The benchmark assesses AI agents on a range of real-world Supabase tasks, including building database schemas, debugging Edge Functions, and fixing Row Level Security (RLS) policies, all within a live, containerized Supabase environment.

Can I use Supabase Evals to test my own AI code generation tools or custom skills?

Yes, Supabase Evals is open-source under Apache-2.0 and designed to run locally, enabling Software Developers to test their own AI agents, custom skills, or even evaluate internal tools against its comprehensive suite of scenarios.