LLM writes TLA+ formal verification: Comparison of 6 major models, Agent automated modeling breakthrough

Core conclusion

ACM SIGOPS released the SysMoBench benchmark, which systematically evaluates the ability of mainstream large models (Claude, GPT, Gemini, DeepSeek, etc.) to write TLA+ formal specifications for real systems. Core findings: LLM has almost full marks for writing TLA+ grammar, but modeling real system behavior is still difficult - and the new generation of Agent tools (Claude Code, Codex) have been able to make breakthroughs in automated modeling, and Specula Agent achieved complete consistency and invariance scores.

Key Points

• Time of incident: 2026-05-10 (article published)
• Evaluation objects: 11 real systems (concurrent synchronization protocol, distributed protocol)
• Core findings: Grammar pass rate is close to 100%, but behavioral consistency is only 30%-60%
• Breakthrough progress: Specula Agent (based on Claude Code/Codex) achieves full modeling for the first time

Background: Why LLM is needed to write TLA+

TLA+ is a language for formal modeling of concurrent and distributed systems. It has been proven to find hidden bugs in complex systems in advance at companies such as Amazon and Microsoft. However, TLA+ has a steep learning curve, which limits its popularity.

If LLM can automatically generate TLA+ specifications from source code, it will revolutionize the threshold for formal verification - engineers only need to describe the properties they want to verify, and AI can automatically complete the modeling work.

But a key question always remains: Does the TLA+ specification written by LLM truly describe the behavior of the system, or is it just reciting a textbook template seen in the training data? **

SysMoBench: four-stage testing from syntax to semantics

SysMoBench splits the assessment of LLM's ability to generate TLA+ specifications into four progressive stages:

Two systemic failure modes

The study found that even if the syntax and runtime pass, LLM-generated TLA+ specifications suffer from two common "textbook modeling" problems:

Pattern 1: State Explosion — The specification allows states that the actual system will never enter. For example, in the specification written by Claude Sonnet for ZooKeeper elections, set union (recvVotes ∪ {newVote}) is used when receiving votes, causing old and new votes to exist at the same time. The actual implementation of ZooKeeper uses the sender ID as the key, and the new vote overwrites the old vote.

Mode 2: Missing state — The specification merges multi-step operations into single-step atomic operations, resulting in some actually reachable states being unreachable in the specification. Also in the ZooKeeper example, LLM checks the global state (epoch comparison) in one step, while the actual code is incrementally completed in multiple steps.

The common source of these two problems: LLM knows "what Raft looks like in the textbook", but does not know "which step of the operation is divided into which sub-steps by Etcd's Raft implementation".

Key breakthrough: Agent automated modeling achieves full score

Different from bare LLM, cutting-edge code agents (such as Claude Code, Codex) show stronger capabilities in TLA+ modeling**. Specula Agent developed by researchers can:

• Automatically read the target code repository
• Judgment of key components of modeling
• Drives a complete specification writing workflow
• Get a perfect score for consistency + immutability on SysMoBench

This means that engineers no longer need to manually guide LLM to write specifications, but instead allow the Agent to autonomously complete the entire modeling process like a human expert. This may be a key step for formal verification to become practical.

What it means for developers

• The threshold for formal verification is lowering: Even if the direct output of LLM is not good enough, with the Agent tool chain, it is feasible to automatically generate verifiable TLA+ specifications.
• The skill boundaries of code agents are expanding: they are no longer limited to writing code and testing, but can also do architecture-level system modeling.
• Focus on consistency when selecting models: If you want to use LLM to assist formal verification, you should choose a model that performs better in the consistency phase of SysMoBench (for example, Gemini 3.1 reaches 81% in the invariance phase)

Run SysMoBench with Agent tool

# Leaderboard
curl -sS https://sysmobench.com | grep -oP '"[^"]*"' | head -20

# Claude Code （）
claude code --prompt "， TLA+ "

• SysMoBench Leaderboard
• Specula Agent （）
• TLA+

AI Claude、Claude Code、GPT、Gemini、DeepSeek、Codex. These tools may have corresponding tool cards maintained on the WayToClawEarn platform side.

Internal link guidance

• Want to get started with the Agent tool quickly? Watch: AI Agent Tools 2026 Complete Tutorial: 5 Tools to Build an Automated Pipeline in 30 Minutes
• Real case: Claude Code one-person company startup: Claude Code 48 hours to start a business: one person + US$29 monthly fee, monthly income in 3 months $9,000
• Want to learn Cluade Code content automation? Watch: Claude Code automated writing practice: build an AI content production pipeline in 30 minutes