Needle's "FFN-free" architecture: Tool calls do not require large models, implications for local AI Agent developers

Core conclusion

On May 12, the Cactus team open sourced Needle, a tool calling model with only 26M parameters, on Hacker News. What is unique about it is that it completely abandons the feed-forward network (FFN) in the standard Transformer and relies only on the attention mechanism to implement tool calls, achieving 6000 tok/s pre-filling and 1200 tok/s decoding speeds on consumer-grade devices.

Key Points

• This is the first public model to verify the hypothesis that "tool invocation does not require FFN" at an ultra-small parameter scale
• Meaning for AI Agent developers: running tools to call models no longer requires a GPU, a mobile phone chip is enough
• Architectural design may rewrite the deployment paradigm of small models - not just quantification, but radically reducing parameters

Background and trigger events

On May 12, Henry from the Cactus team released Needle on Hacker News, a 26M parameter model designed for function calls. The post received 445 points, ranking among the top three most popular HN posts of the day.

The Cactus team's core observation is that tool calls are essentially "retrieval and assembly", not inference. Matching query to tool name, extracting parameter values, and assembling JSON—these three steps are all about aligning and copying between input and output, which is what cross-attention is best at. No step requires position-wise feature transformation (the core functionality of FFN).

Based on this observation, they designed a "Simple Attention Network": the entire model only has self-attention and cross-attention layers, without any MLP/FFN. The experimental results verified the hypothesis - the 26M Needle surpassed the 270M FunctionGemma, 600M Qwen, 350M Graninte and other models with larger parameter sizes in a single function call task.

Key Impact (by Dimension)

Why "No FFN" is important for AI Agent development

About 2/3 of the parameters of traditional Transformer come from FFN. For models smaller than 50M, these parameters contribute far less to structured tasks (such as tool calls) than more attention layers. After removing FFN:

1. The number of parameters is reduced by 2/3 - directly reducing the delay bottleneck (memory bandwidth) on edge devices
2. softmax itself is nonlinear——softmax(QK^T/sqrt(d)) * V is a data-dependent nonlinear mixing operation, which is sufficient for routing information
3. The encoder-decoder structure is naturally suitable for tool calls - tool definitions are structured objects, and bidirectional encoders can see the complete definition at once; causal models need to reason from left to right about the structure

This is particularly relevant to the WayToClawEarn community: if you are using local models to build AI Agents (such as using OpenClaw for content automation, using n8n + local LLM for Agent workflow), Needle has proven a clear path - the tool call model can be compressed to 26M and run on mobile phone chips without expensive GPU inference.

Adaptation suggestions

1. Run Needle on your own device: After cloning the warehouse, directly use Hugging Face weight inference, and it can be run on Mac/PC.
2. Fine-tune your toolset: Synthesize your tool-call dataset with Gemini or other large models and fine-tune it on Needle
3. Architecture reference: If you train a small model by yourself, consider removing the FFN layer and replacing it with deeper attention.
4. Integration with local AI Agent framework: Use Needle as a tool calling engine for OpenClaw and Hermes Agent

# Needle

# : https://huggingface.co/Cactus-Compute/needle

# Mac/PC
git clone https://github.com/cactus-compute/needle.git
cd needle && pip install -r requirements.txt

# query + tools
python -c "
from needle import Needle
model = Needle.from_pretrained('Cactus-Compute/needle')
tools = [
    {'name': 'send_email', 'params': {'recipient': 'string', 'subject': 'string'}},
    {'name': 'set_timer', 'params': {'duration': 'int'}}
]
result = model.call('5', tools)
print(result)  # {'name': 'set_timer', 'params': {'duration': 300}}
"

Tool entry

The models and tools involved in this article are: Gemini (data synthesis), OpenClaw (Agent framework), n8n (workflow automation), Hermes Agent (AI agent framework), DeepSeek (open source model), ChatGPT (data synthesis alternative)

Internal link guidance

• Want to learn how to run small models locally? See: How to run local AI models on M4 Mac with LM Studio: A complete 30-minute tutorial
• Want to know how to call the AI Agent tool? See: AI Agent-Driven Content Automation: n8n MCP Building Guide from Scratch
• Real case: He used AI Agent to build a SaaS with a monthly income of $5,000: 18-Year-Old Built a $5,000/mo SaaS With AI Agents — Zero Hand-Written Code