SWE-bench Verified is saturated: OpenAI announces that this coding benchmark no longer reflects cutting-edge AI capabilities

Core conclusion

On April 27, 2026, OpenAI issued an announcement announcing that the SWE-bench Verified benchmark is no longer suitable for measuring the coding capabilities of cutting-edge AI models. This news generated 298 hot comments and 165+ discussions on Hacker News. In the past two years, mainstream AI coding models have improved from 30% to nearly full scores on this benchmark, and SWE-bench's differentiation has basically disappeared. **What this means for the field of AI coding: the old era is over, and new evaluation standards are on the way. **

Key Points

• Time of incident: April 26-27, 2026 -Affected objects: All teams and individual developers using AI coding tools
• Core change: SWE-bench Verified is no longer a reliable indicator of differentiating cutting-edge coding models

Background and trigger events

SWE-bench is a software engineering benchmark launched by the Princeton University team to evaluate the ability of AI models to solve GitHub Issues in real code warehouses. In 2025, OpenAI partnered with Princeton to launch SWE-bench Verified as a curated subset of the benchmark.

Over the past 18 months, AI coding capabilities have experienced explosive growth:

• End of 2024: Top models score around 40-50% on SWE-bench Verified
• Mid-2025: Claude Code, GPT-5, etc. reach 70-80%
• Early 2026: Frontier models generally break through 90%, and some are close to 95-98%

When all competitors are close to the ceiling, the benchmark itself naturally loses its evaluation value. OpenAI’s official blog post states that new, more challenging coding assessment frameworks are needed to continue to drive and measure the evolution of AI coding capabilities.

Key Impact

Adaptation suggestions

For AI coding tool users

• Don’t just look at the benchmark to select tools: When all models are close to 95%, the benchmark scores have no actual reference value. You should return to real task testing: let the actual performance of Claude Code, ChatGPT, Cursor and other tools in your own project speak for itself.
• Focus on workflow integration: The value of AI coding does not lie in a single code generation, but in the automation level of complete workflows such as PR Review, CI/CD integration, test generation, etc.
• Establish an internal evaluation system: If your team relies on AI coding tools, it is recommended to establish its own evaluation data set (based on Issue fix tasks in the actual code base) rather than relying on third-party benchmarks.

For AI coding tool developers

• Migrate to the new benchmark as soon as possible: If your product still uses SWE-bench Verified as the main promotional indicator, you need to start looking for or participating in a new generation of benchmark tests.
• Emphasis on actual output indicators: Real business indicators such as user retention rate, code adoption rate, and development efficiency improvement are more convincing than benchmark scores.

Example: Custom testing framework for evaluating coding AI

# AI
import json
import subprocess

def evaluate_ai_coding(tool_name: str, test_cases: list[dict]) -> dict:
 """ AI """
    results = []
    for case in test_cases:
        prompt = case["prompt"]
        expected_output = case["expected"]

        actual_output = subprocess.run(
            ["claude", prompt],
            capture_output=True, text=True
        ).stdout

        results.append({
            "case": case["name"],
            "passed": evaluate_output(actual_output, expected_output),
            "execution_time": 0
        })
    return results

Related extended information

• Hacker News
• SWE-bench
• OpenAI ：SWE-bench

Tool entry

OpenAI announced that SWE-bench Verified is no longer suitable for measuring cutting-edge coding capabilities, but OpenAI’s ChatGPT and Claude are still among the strongest coding AI tools available. It is worth noting that Claude Code performs well in practice as a coding agent, even without relying on some task types defined by SWE-bench. Open source models such as DeepSeek are also rapidly catching up on the coding circuit. Orchestration frameworks like LangGraph allow AI Agents to handle complex coding tasks more reliably.

Internal link guidance

• Want to build an AI coding workflow yourself? See: AI Agent Tools 2026 Complete Tutorial: 5 Tools to Build an Automated Pipeline in 30 Minutes
• Real case: Someone used Claude Code to earn $9,000 per month in 48 hours, see: Claude Code 48 hours to start a business: one person + US$29 monthly fee, monthly income in 3 months $9,000