Open source 26M parameter model Needle released: distilling Gemini tool calls into consumer-grade devices

Core conclusion

On May 13, 2026, the Cactus team released Needle - an open source tool calling (function calling) model with only 26M parameters on Hacker News. By eliminating the FFN (feedforward network) layer in traditional Transformers, it proves that tool calls are not essentially inference tasks, but retrieval-assembly tasks. This means that AI Agents running on mobile phones, watches, and smart glasses no longer require large models.

Key Points

• Published: 2026-05-13 (Hacker News debut)
• Model size: 26M parameters, only attention layer + gating mechanism, no MLP
• Performance Data: 6000 tok/s prepopulation, 1200 tok/s decode on consumer devices
• Training Cost: 200B tokens pre-training (27 hours on 16x TPU v6e) + 2B tokens post-training (45 minutes)
• Open Source License: MIT protocol, weights can be downloaded from Hugging Face

Background: Why does the Agent model need a small model?

The current AI Agent ecosystem faces a fundamental contradiction: Agent experience requires real-time response, but large models are too heavy to run on end-side devices.

Although models such as OpenAI's GPT-4o and Anthropic's Claude Opus have strong tool calling capabilities, they have high inference costs and large delays. The performance of small models (<1B parameters) on tool calling tasks has been unsatisfactory.

The Cactus team’s insight was: tool invocation = retrieval + assembly, not inference. The model doesn't need to remember a lot of world knowledge (that's FFN's job), it just needs to match tool names based on user intent, extract parameters, and output JSON. Cross-attention is the right primitive to do this.

Needle’s key innovation: Simple Attention Networks

Needle's core architectural innovation is Simple Attention Networks (SAN):

This finding is general: Any task that relies on external structured knowledge (tool invocation, RAG, retrieval augmentation generation) does not require the model to "remember" the facts in FFN. The model only needs to learn to match and assemble if the facts are provided in the input.

Performance comparison

In the single function call benchmark test, Needle (26M) performance:

Needle beats similar purpose-built models with 10-20x fewer parameters.

Training process: low cost + synthetic data

Needle's training process is worthy of attention because it demonstrates extremely low training costs:

1. Pre-training phase: Pre-training on 200B tokens, using 16 TPU v6e, taking 27 hours
2. Post-training phase: Fine-tuning on synthetic function call data of 2B tokens, only 45 minutes
3. Data synthesis: Training data covering 15 tool categories (timers, messages, navigation, smart home, etc.) was generated through Gemini

The entire training process can be completed in less than 30 hours, which is an extremely low threshold for teams that want to customize their own dedicated tool-calling models.

How to run Needle locally

Needle has been released on GitHub and Hugging Face, and can be run and fine-tuned directly on your Mac/PC:

#
git clone https://github.com/cactus-compute/needle
cd needle

#
pip install -r requirements.txt

#
python run.py --model Cactus-Compute/needle

Cactus Playground 。

Needle AI /

WayToClawEarn ，Needle

1. Agent

26M ** App、、**。AI Agent GPT-4o， 26M 。

• API （ API ）
• （）
• （）

2.

n8n / Make.com ， Needle ****，、—— OpenAI/Claude API 。

3.

， VPS Raspberry Pi 。** Agent **，。

Gemini、OpenAI、Claude、Claude Code、DeepSeek、n8n、Make.com、Hugging Face.

Next steps: Content you may be interested in

Want to run local models on your Mac? Check out this tutorial:

• How to run local AI models on M4 Mac with LM Studio: A complete 30-minute tutorial

Want to know how to use the AI Agent tool?

• AI Agent Tools 2026 Complete Tutorial: 5 Tools to Build an Automated Pipeline in 30 Minutes

Real case: An 18-year-old with zero foundation used AI Agent to create a SaaS with a monthly income of $5,000:

• 18-Year-Old Built a $5,000/mo SaaS With AI Agents — Zero Hand-Written Code

*This article is based on the Cactus team’s open source release on Hacker News (2026-05-13) and the content of the GitHub repository. *