Google Gemma 4 launches multi-Token prediction: AI inference speed is increased by 3 times, and the cost is further reduced

Core conclusion

Google officially released Gemma 4's Multi-Token Prediction (MTP) technology on May 5, 2026. By introducing a small "draft model" for speculative decoding, the inference speed is increased by up to 3 times while maintaining the same output quality. The technology is already available in Gemma 4 series models and is fully open source to developers.

Key Points

• Release time: 2026-05-05
• Core changes: Gemma 4 introduces the MTP draft model, speculative decoding achieves 2-3 times inference acceleration
• Affected objects: AI application developers, automated workflow builders, self-hosted inference users
• Available Status: Integrated into Gemma 4 model, community adaptation is in progress

Background: Inference speed bottleneck

As the parameter scale of large language models continues to grow, inference speed has become the main bottleneck restricting the implementation of AI applications. Traditional autoregressive decoding generates one token each time and cannot fully utilize the parallel computing capabilities of modern hardware. For developers running AI automation workflows, inference latency directly impacts responsiveness and operational costs.

Since its release, Google's Gemma 4 open source model series has been favored by developers for its excellent performance. However, in actual use, the inference speed of Gemma 4 still lags behind competing models such as Qwen. Several users mentioned in the Hacker News discussion that speed was the main reason for choosing Qwen over Gemma 4.

Multi-Token prediction technology analysis

The core idea of MTP is to use a lightweight "draft model" to predict multiple subsequent Tokens at once, and then the main model verifies the correctness of these predictions. This method of speculative decoding allows the model to take advantage of the parallel computing capabilities of modern GPUs to generate multiple Tokens simultaneously in a single forward pass.

The key advantage is: MTP does not sacrifice quality for speed. After the draft model generates candidates, the main model verifies them—if the candidate is correct, it is accepted; if it is wrong, the main model corrects it. So the final output is exactly the same as pure autoregressive decoding.

The technical route adopted by Gemma 4 is similar to NVIDIA Nemotron's MTP solution, but in a speculator-style manner, that is, additional draft models are run as plug-ins instead of directly baking multi-Token predictions into the main model.

Impact on AI automated workflows

For developers running AI Agents and automated processes, the significance of MTP technology is reflected in three levels:

1. The cost of reasoning is directly reduced A 2-3x increase in inference speed means more requests can be processed in the same time, or the same workload can be completed with fewer computing resources. For self-hosted scenarios, this translates directly into reduced GPU costs.

2. Shortened response delay When calling Gemma 4 models in automated tools such as n8n, Claude Code, OpenClaw, etc., MTP reduces the wait time for each inference by more than half, making real-time interactive workflows feasible.

3. Local deployment is more practical The community has added MTP support for Qwen models in llama.cpp, and Gemma 4 support is on the way. This means that the inference experience on consumer-grade hardware will be a qualitative leap—a single user can obtain a smooth experience of 100+ TPS locally.

Adaptation suggestions

If you are already running Gemma 4 models locally or in the cloud, the following steps can help you quickly take advantage of MTP technology:

1. Check model version: Confirm that the Gemma 4 version you are using supports MTP (the official documentation has been updated)
2. Download draft model: Google provides a matching small drafter model (<1B parameters)
3. Updated inference framework: Pay attention to updates to llama.cpp and LM Studio, which are adding MTP support
4. Benchmark Test: Test the acceleration effect under actual workloads. Different tasks may have different benefits.
5. Monitor memory: Although the draft model has additional memory overhead, the impact is usually controllable.

Related extended information

• Google AI Blog: Multi-token-prediction in Gemma 4
• Hacker News : Accelerating Gemma 4

Tool entry

The following tools have appeared naturally in the text: Gemini, Gemma, Gemini, Claude Code, n8n, OpenClaw, llama.cpp, LM Studio

Internal link guidance

• Want to use AI Agent to build automated workflow? Watch: AI Agent Tools 2026 Complete Tutorial: 5 Tools to Build an Automated Pipeline in 30 Minutes
• Practical solution for local inference acceleration: Claude Code + DeepSeek V4 Building Tutorial: API Fees Dropped by 90% (15 Minutes)
• Real-life examples of automating content with AI: 18-Year-Old Built a $5,000/mo SaaS With AI Agents — Zero Hand-Written Code