Running a 70B large model locally is not a dream: Intel AutoRound quantitative practical tutorial

One sentence summary

Intel AutoRound is an open source large model quantification tool that can compress a large model with 70B parameters to 2-4 bits. It can run smoothly on a consumer-grade graphics card (single RTX 4090) with minimal performance loss.

After reading this article you will know:

• What is model quantization and why do you need it
• Use AutoRound to compress the 70B model to run on 24GB of video memory
• Quantified performance loss (including actual measured data)
• Comparison with GPTQ and AWQ

What is model quantization?

Parameters for large language models are typically stored in FP16/BF16 precision (2 bytes per parameter). A 70B parameter model requires approximately 140GB of video memory.

Quantization is to reduce the parameter precision to 4-bit (0.5 bytes/parameter) or lower:

• FP16 70B → 140GB
• 4-bit 70B → 35GB
• 2-bit 70B → 17.5GB

This means you can run the 70B model on a single RTX 4090 (24GB)!

AutoRound vs GPTQ vs AWQ comparison

The advantage of AutoRound is that it does not require a calibration data set (required by GPTQ) and the quantization speed is 3-5 times faster than GPTQ.

Practical combat: Use AutoRound to quantify DeepSeek V4

Environment preparation

pip install auto-round transformers torch

GPU 24GB （RTX 3090/4090/A5000 ）。

from auto_round import AutoRound
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "deepseek-ai/deepseek-v4"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# 4-bit
bits = 4
group_size = 128

model_quantized = AutoRound.quantize(
    model,
    tokenizer=tokenizer,
    bits=bits,
    group_size=group_size,
    device="cuda:0"
)

#
model_quantized.save_pretrained("./deepseek-v4-4bit")
tokenizer.save_pretrained("./deepseek-v4-4bit")

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "./deepseek-v4-4bit",
    device_map="auto"
)

DeepSeek V4（70B）

****4-bit 1-2 ， 140GB 36GB， 3 。，。

Tool entry

DeepSeek, NVIDIA, vLLM.

Next action

1. If you have an RTX 3090/4090, try quantizing DeepSeek V4 today
2. Do you want to use it directly without quantification? Call DeepSeek V4 API with OpenRouter (10x cheaper than GPT-5)
3. Need stronger reasoning skills? Try DeepSeek R2 (1.7T parameters)

Related reading

• DeepSeek R2 full review: 1.7T parameters vs GPT-5 vs V4

• DeepSeek V4 vs Claude Code actual test comparison

• Claude Code + DeepSeek V4 building tutorial

• DeepSeek R2 Review: 1.7T Params vs GPT-5 Benchmarks

• DeepSeek Reasonix in action: Build an AI programming agent at zero cost (30-minute tutorial)

• DeepSeek V4 vs Claude Code: 90% Cheaper, Same Quality

• Claude Code + DeepSeek V4 Building Tutorial: API Fees Dropped by 90% (15 Minutes)