Unsloth teamed up with NVIDIA to optimize LLM fine-tuning: training speed increased by 25%, memory almost unchanged

Core conclusion

On May 6, 2026, Unsloth released a joint optimization announcement with NVIDIA. The two teams focused on the "hidden bottlenecks" that are easily overlooked in LLM fine-tuning - not the recognized computational hot spots such as matrix multiplication or attention mechanisms, but three types of "meta-information overhead" such as packaged sequence metadata reconstruction, gradient checkpoint serialization and MoE routing redundant calculations.

Key Points

• Event time: 2026-05-06 -Affected objects: All developers/teams who use Unsloth for LLM fine-tuning
• Core changes: After the three-stage optimization is superimposed, the training speed is increased by about 25% without sacrificing model accuracy.
• Applicable hardware: NVIDIA RTX series to B200 Blackwell full line of GPUs

Background: When mainstream computing is no longer a bottleneck

When developers optimize fine-tuning performance, they usually give priority to high-impact computing cores such as matrix multiplication, attention mechanism (Attention), fused operators (Fused Ops), and grouped GEMM. But Unsloth and NVIDIA engineers found that when these mainstream calculations were fully optimized, a different kind of bottleneck began to emerge - the GPU stalled on metadata-dependent work.

Specifically, there are three types of modes:

Detailed explanation of three core optimizations

1. Cache packaged sequence metadata

When multiple short sequences are packed into one long sequence for training, the model needs to know the start and end positions of each original sequence. This produces a set of metadata: sequence length, cumulative offset, maximum sequence length, and attention mask structure.

Key Insight: For a fixed packaging batch, these metadata are the same at every layer. If the model has L layers, the traditional approach is to reconstruct this information at each layer - doing the same work L times.

What Unsloth does is to cache the packaging metadata of the current batch and reuse it across layers. Actual measurement on Qwen3-14B QLoRA SFT:

• Forward propagation speed increased by 43.3%
• Backpropagation speed increased by 5.8%
• Total speed increase per batch 14.3%

The core benefit of caching is that it removes duplicate coordination work on hot paths. Forward propagation benefits the most because this is the most intensive stage of metadata repeated consumption.

2. Double buffered gradient checkpoint

Standard Activation Checkpointing saves video memory by not saving all intermediate activations. But the problem is the data flow in default mode: copy activation from CPU to GPU → wait for copy to complete → perform reverse computation on that activation → start next layer of copy.

Unsloth introduces a double buffering mechanism: while backpropagation runs on buffer A, the copy stream can preload the next activation into buffer B. This way copying and computation are done in parallel instead of waiting serially.

Actual measurement on NVIDIA B200 Blackwell GPU:

• 8B model: +8.40% speed increase, only 0.37 GB more video memory
• 14B model: +6.70% speed increase, only 0.47 GB more video memory
• 32B model: +4.61% speed increase, only 0.23 GB more video memory

Benefit logic: The larger the model, the higher the hidden dimension, the greater the amount of data movement, and the more significant the effect of hiding replication delays.

3. GPT-OSS MoE routing optimization

The routing operation of the MoE (mixed expert) model - deciding which tokens are sent to which expert - uses the argsort + bincount grouping strategy. Complete routing calculations and data rearrangement are re-performed at each layer before.

After optimization, tokens are grouped through one argsort, and then each layer directly reuses the grouping results to avoid repeated calculations. This optimization has the most significant impact on MoE architectures (e.g., DeepSeek MoE, Mixtral).

Adaptation suggestions

If you are already using Unsloth for LLM fine-tuning, the above optimizations will take effect completely automatically - just upgrade to the latest version without modifying any training code. If you are new to Unsloth, this is a good time to get started:

• Use Unsloth to fine-tune large models, and local RTX 4090 can run QLoRA training with 7B~14B parameters
• The newly optimized cache and double buffering mechanism significantly improves the training throughput and further reduces the unit computing cost.
• For independent developers, AI agent tool users, and content automation practitioners, this means a lower threshold for model fine-tuning

Want to learn how? See: DeepSeek V4 vs Claude Code: 90% Cheaper, Same Quality

Related extended information

• Unsloth

Tool entry

Unsloth, NVIDIA, DeepSeek, Claude Code, MoE and other terms appearing in the text are automatically matched by the platform side to the maintained tools library, and tool_mentions floating cards are generated.

Internal link guidance

• Want to save on API fees? Watch: DeepSeek V4 vs Claude Code: 90% Cheaper, Same Quality
• Real cases of making money with AI tools: A real case of a data analyst using Claude Code + n8n to build an automated report SaaS with a monthly income of $3,800