Forge open source release: Guardrail system allows 8B local model to jump from 53% to 99% in Agent missions

Core conclusion

Forge is an open source LLM reliability layer designed for tool invocation scenarios of self-hosted models. Its core findings are astonishing: Without the guardrail system, the 8B parameter local model only achieved a 53% success rate in Agent multi-step tasks; with Forge's guardrail system added, the same model soared to 99.3%. This means that an 8B local model running on $600 GPU almost equals the performance of leading cloud models in Agent tasks.

Key Points

• Event: Texas Instruments AI Director Antoine Zambelli releases open source project Forge
• Goal: Solve the reliability problem of self-hosted LLM in multi-step Agent tasks
• Core numbers: 53% → 99.3% for 8B model, 100% for Claude Sonnet + Forge
• Verification: Accepted into ACM CAIS '26, presented in San Jose, May 26-29
• Applies to: Ollama, llama-server, Llamafile and Anthropic backends

Background: Agent dilemma of 8B model

Self-hosted local models have always faced a mathematical challenge: multi-step error accumulation. For a 5-step Agent workflow, if each step has an accuracy of 90%, the overall success rate is only about 59%. In reality, the accuracy of each step of small models is often lower.

Existing Agent frameworks (LangGraph, CrewAI, etc.) are optimized for cutting-edge models in the cloud and do not provide systematic solutions for the inherent weaknesses of local models - error step recovery, malformed tool calls, context window management. Forge directly targets this gap and solves it through a three-layer guardrail system: response parsing repair, step execution enforcement, and VRAM-aware context management.

Key Impact (by Dimension)

Adaptation suggestions

Forge provides three usage modes:

WorkflowRunner — Completely define tools, select backends, and run structured Agent loops. One-stop management of system prompts, tool execution, context compression, and guardrail systems. Suitable for scenarios built directly on top of Forge.

Guardrails Middleware — Use Forge’s reliability stack as composable middleware inside your orchestration loop. You control the loop and Forge validates responses, fixes malformed tool calls, and enforces necessary steps.

Proxy Server — The simplest way. python -m forge.proxy Launches an OpenAI compatible agent to transparently apply the guardrail system. Existing clients (including OpenCode, Continue, aider, etc.) will benefit without modification.

Task List

• Add Forge's three-layer guardrail to the self-hosted Agent: response parsing repair, step execution enforcement, and context compression
• Benchmark your model/backend combination using Forge Eval Harness
• Prioritize deploying self-hosted solutions instead of public APIs in privacy-sensitive scenarios

Related extended information

• YouTube Demo:
• GitHub: github.com/antoinezambelli/forge
• ACM CAIS '26: San Jose, May 26-29

Tool entry

Forge supports Ollama, llama-server (llama.cpp), Llamafile and Anthropic backends. Tool names that appear naturally in the text will automatically match the tool floating card on the platform side.

Next action

This project demonstrates an important trend: The bottleneck of Agent reliability has shifted from model capabilities to system architecture. If you are building an AI automated workflow, you may wish to first evaluate the lack of guardrail systems in your pipeline - this may be the most cost-effective improvement point.

• Want to learn how to add quality gates to AI automated workflows? Watch: How to add quality gates to your AI automation workflow: A practical guide from output to trustworthy results
• Want to run a local large model? Watch: How to run local AI models on M4 Mac with LM Studio: A complete 30-minute tutorial
• Real case: He used Claude Code + AWS to build AI SaaS: He used Claude Code + AWS to build AI SaaS, and his monthly income was $12,000 for 3 months