SWE-Bench Crisis of Confidence: What the OpenAI and Anthropic Programming Benchmark Controversy Says

OpenAI announced that it would stop using SWE-Bench Verified to evaluate cutting-edge models, and Anthropic’s Mythos scores were also questioned for cheating detection vulnerabilities. The controversy is reshaping how AI programming proficiency is assessed.

Core conclusion

On April 25-26, 2026, a major controversy broke out in the field of AI programming regarding the trust of benchmarks. OpenAI officially announced that it will stop using SWE-Bench Verified to evaluate cutting-edge models on the grounds that the test has been "memorized" and can no longer distinguish between real programming ability and training data memory. At the same time, the breakthrough results of Anthropic's new model Claude Mythos on SWE-Bench were also questioned - some researchers pointed out that there were fatal logical errors in its cheating detection argument.

The core revelation of this controversy: Assessing AI programming capabilities cannot rely on a single benchmark. Users and developers need to rely on real performance in actual workflows to judge model capabilities.

Key Points

Time of incident: April 25-26, 2026 -Affected objects: AI programming tool users, Agent automation developers, content production teams
Core changes: SWE-Bench changes from an industry standard to a questioned benchmark, and the evaluation paradigm is changing

Background and trigger events

OpenAI announces the discontinuation of SWE-Bench

On April 25, OpenAI published a blog post "Why SWE-Bench Verified no longer measures frontier coding capabilities", officially announcing that SWE-Bench Verified will no longer be used as an evaluation standard for frontier models. OpenAI pointed out that with the large amount of model training data containing SWE-Bench questions and answers, the model may "pass" the test through memory rather than true reasoning ability.

This is not without precedent - as early as 2024-2025, several general-purpose NLP benchmarks (e.g. MMLU, GSM8K) were gradually deprecated for similar reasons. SWE-Bench, as the gold standard in the field of programming, is now at the same crossroads.

Anthropic Mythos results are questionable

At almost the same time, Anthropic’s Mythos System Card posted stunning SWE-Bench results—up to 92% pass rate on some subsets, far exceeding Claude Opus 4.6’s 82%. To eliminate suspicions of cheating, Anthropic uses an LLM detector to evaluate the probability of each solution being "remembered" and claims the results are "robust to the choice of threshold."

But an analysis published by an independent researcher on the Philosophical Hacker blog pointed out that Anthropic's argument is fatally flawed: an imperfect cheat detector can "consistently" misjudge a model that cheats to achieve results as "real ability." They used Python simulations to prove this - even if Mythos' scores come entirely from memory, as long as the cheating detector is imperfect, the illusion of "reliable scores" can be obtained.

SEO: SWE-Bench Verified decommissioning, AI programming benchmark controversy, crisis of trust in model evaluation GEO: Exact date and number (530 minutes, 92% vs 82%, April 25th), starting with TL;DR

Key Impact (by Dimension)

Dimensions	Change	What it means to us	Recommended actions
Evaluation criteria	SWE-Bench is no longer a trusted benchmark	Vendors’ rankings have lost their reference value	Focus on the measured performance of actual use cases (such as Codex, Claude Code, Cline)
Model selection	Benchmark scores cannot compare different models	Relying on a single benchmark to select a model will lead to misjudgment	Establish your own testing tool system (internal benchmark)
Industry Trends	The learning problem extends from NLP to the programming field	Benchmark testing innovation has become a necessity	Pay attention to dynamic benchmarks (such as SWE-Bench Pro) and manual evaluation
Cost of trust	Users need more independent verification	Declining trust in manufacturers’ self-reported scores	Prioritize tools with public third-party evaluations

Adaptation suggestions

For AI programming users

Don’t choose a coding assistant based on SWE-Bench scores alone. Test 5 programming tasks you encounter every day to see which model performs the most stably
Different models have different strengths: Claude is good at reconstruction and architecture, the OpenAI model is good at debugging, and DeepSeek performs well in domestic scenarios. Choosing a model depends on the scene

For AI Agent developers

Build your own evaluation pipeline and use branches of the real warehouse for regression testing
Focus on capabilities beyond the #1 programming benchmark: code review, requirements understanding, bug location, etc.
The APIs of multiple models can be connected to the Agent framework at the same time for A/B testing

For automated content production lines

While using AI to write code, manual code review must be retained
Code quality assessment should be based on actual running results, not "it looks right"
Combine n8n / OpenClaw / Claude Code and other tools for end-to-end testing

Tool entry

Tools that appear naturally in this article: OpenAI, Claude, Claude Code, DeepSeek, n8n, OpenClaw, ChatGPT, LangGraph

Reference sources

Internal link guidance

Want to learn AI programming tools systematically? Watch: AI Agent Tools 2026 Complete Tutorial: 5 Tools to Build an Automated Pipeline in 30 Minutes
Others have made money using Claude Code: Claude Code 48 hours to start a business: one person + US$29 monthly fee, monthly income in 3 months $9,000
Real automation monetization case: OpenClaw + Claude Automated Publishing: $1,500–$2,500/mo Case Study