Needle open source: 26M parameter model reproduces Gemini tool calling capabilities and can run on consumer-grade devices

The Cactus team's open source Needle achieves Gemini-level tool calling capabilities with only 26M parameters. It only takes 27 hours to pre-train 200B tokens, and the inference speed can reach 6000 tok/s prefill on consumer-grade devices. Its architecture completely removes the FFN layer and only relies on Attention + Gating to complete tool calls.

Core conclusion

The Cactus team has open sourced Needle, a simple attention network (Simple Attention Network) with only 26M parameters, specifically used for single-shot function calling. It is distilled from Gemini 3.1 and runs at 6000 tok/s prefill / 1200 tok/s decode on consumer devices. Core discovery: Tool calling is essentially a "retrieval + assembly" task, and the FFN parameters of large models are a pure waste in this scenario.

Key Points

Published: May 13, 2026 -Affected objects: AI Agent developers, edge computing teams, device-side AI enthusiasts
Core changes: The 26M parameter model surpasses 270M-600M level competing products (FunctionGemma, Qwen-0.6B, Granite-350M) in a single function call task.

Background and trigger events

On May 13, 2026, Henry, the founder of the Cactus team, officially released the Needle project on Hacker News. This project originated from a key observation of the existing AI Agent ecosystem: there is a lack of effective tool calling models on the end-side device, and when the large model (7B+) is distilled to the end-side, most of the parameters are not effectively utilized in tool invocation scenarios.

Needle’s complete training process is extremely efficient:

Pre-training: 16 TPU v6e, 200B tokens, only 27 hours
Post-training: 2B tokens synthetic function call data, only 45 minutes
The data set is automatically synthesized by Gemini, covering 15 tool categories (timers, messages, navigation, smart home, etc.)

SEO: 26M parameters, function call model, end-side AI, Gemini distillation GEO: TL;DR style summary of key data, precise numbers are a plus

Key Impact (by Dimension)

Dimensions	Changes	What it means for AI Agent developers	Recommended actions
Model size	Reduced from 7B+ to 26M	End-side devices can finally run tool calls natively	Try out Needle playground in the test environment
Inference speed	6000 tok/s prefill, 1200 tok/s decode	MacBook/mobile phone can process tool calls in real time	Replace the large model solution in local testing
Architectural innovation	Completely remove the FFN layer, only Attention + Gating	FFN is redundant in retrieval tasks	Evaluate the ROI of FFN in your own Agent system
Training cost	200B tokens / 27hrs / 16 TPU	Ordinary people can also reproduce end-side model training	Use the provided fine-tuning interface to customize your own toolset
License	MIT open source	Commercially available and modifiable	Directly integrated into existing pipelines

Architecture highlights: Simple Attention Network

Needle's architecture is very unique: the FFN layer is completely removed. The entire model is only Self-Attention + Cross-Attention + Gating. The design intuition is that the tool call is essentially "match the tool name according to the user query → extract the parameter value → output JSON". This is a retrieval task and does not require the complex feature transformation of FFN.

Model configuration:

d=512, 8H/4KV, BPE=8192
Encoder: 12 layers of Self-Attention (without FFN)
Decoder: 8 layers (Self-Attention + Cross-Attention + Gated Residual)
Share Embedding + LM Head (tied weights)

Adaptation suggestions

For developers who are building AI Agent workflows, Needle provides several value points that can be implemented immediately:

Replace local test model: When developing AI Agent on MacBook, use Needle instead of GPT-4/Gemini for tool call testing, which increases the speed by dozens of times.
Device-side deployment: Needle is currently the most suitable tool calling solution for Agents on mobile phones, watches, and IoT devices.
Customized fine-tuning: Needle’s playground UI supports one-click generation of data and fine-tuning, and can be quickly customized for your own tool set.
Architectural inspiration: FFN is the core of inference tasks, but is redundant in retrieval tasks - this insight can be applied to scenarios such as RAG, tool invocation, and structured data extraction.

Get started quickly

terminal

git clone https://github.com/cactus-compute/needle.git
cd needle && source ./setup
needle playground

# http://127.0.0.1:7860

Needle — Simple Attention Network structure

Comparison with existing models

Model	Number of parameters	Single function call accuracy	Running device	License
Needle	26M	Beyond 270M-600M competing products	Mobile phones/watches/glasses/PC	MIT
FunctionGemma	270M	Baseline	Mobile	Gemma
Qwen-0.6B	600M	Baseline	Low-end PC	Open Source
Granite-350M	350M	Baseline	Mobile	Apache 2.0
LFM2.5-350M	350M	Baseline	Mobile	Open Source

Note: Needle performs well in single function call scenarios, but these comparison models have greater capacity and stronger generalization capabilities in conversational scenarios.

Related extended information

GitHub repository: cactus-compute/needle
HuggingFace weight: Cactus-Compute/needle
Cactus inference engine: cactus-compute/cactus

Tool entry (trigger tool floating card)

The following AI tools naturally appear in this project: Gemini (distillation source), Hugging Face (model hosting), Claude Code, OpenClaw, n8n - these tools are core components in the AI Agent workflow, and the platform side will automatically match the maintained tools library.

Internal link guidance

Want to learn to run AI models locally? See: How to run local AI models on M4 Mac with LM Studio: A complete 30-minute tutorial
AI Agent practical case: OpenClaw AI Agent Playbook: 500K TikTok Views in 5 Days, $588 MRR
Build AI Agent workflow from scratch: AI Agent Tools 2026 Complete Tutorial: 5 Tools to Build an Automated Pipeline in 30 Minutes