HBM memory accounts for 63%: AI chip cost structure has reversed, how developers should respond

Core conclusion

The latest data from Epoch AI reveals fundamental changes in the cost structure of AI chips: High-bandwidth memory (HBM) has grown from an auxiliary component to the absolute cost main body of AI chips, accounting for 52% in Q1 of 2024 to 63% in Q4 of 2025. During the same period, the proportion of advanced packaging (CoWoS) dropped from 19% to 15%, auxiliary components dropped from 15% to 9%, and the proportion of logic chip die stabilized at around 13%.

Key Points

• Surge in total volume: The total spending on AI chip components by the four major chip vendors (Nvidia, AMD, Google, Amazon) will increase from US$22 billion in 2024 to US$52 billion in 2025, and HBM alone will contribute approximately US$20 billion to the increase.
• Price pressure continues: HBM supply remains tight in 2026. About US$25 billion of Microsoft's US$190 billion capex in FY2026 comes from component price increases. Meta also raised the 2026 capex range by US$10 billion due to component price increases.
• Impact on developers: The cost structure of AI inference and training is repriced by the upstream memory market, and the cost-efficiency criteria for selecting models are changing

Background: HBM becomes the "new oil" for AI chips

High-Bandwidth Memory is the core supporting component of AI chips. Mainstream AI accelerators such as Nvidia H100/B200, AMD MI300X, Google TPU v5, etc. all rely on HBM to provide the ultra-high bandwidth and capacity required for model parameters.

Epoch AI's analysis covers the four major AI chip designers, Nvidia, AMD, Google, and Amazon. It estimates the single-chip cost of each type of component weighted by production volume, and then multiplies it by quarterly shipments to get the total expenditure. This dimension can better reflect the resource flow of the entire industry than simple material costs.

The key turning point occurs in 2024–2025: the transition from HBM3 to HBM3E, the expansion of memory capacity lagging behind the growth of AI chip shipments, and the increase in pricing power of HBM suppliers (SK Hynix, Samsung, Micron) have jointly promoted the continued rise of HBM's share.

Practical impact on AI developers

1. Long-term pressure on API pricing

The rising cost of HBM will eventually be passed on to API call prices. Nvidia H100 cloud instance rental prices have already seen a 10-15% increase in Q3 2025, mainly from the increase in the cost of memory components. If you are using GPT-4, Claude or DeepSeek’s API, you may see more price increase notifications in the next 6-12 months.

2. Changes in local model deployment cost structure

For teams that build their own AI inference infrastructure, the increase in the cost ratio of HBM means that the premium for GPU memory capacity specifications is increasing - the cost performance of SKUs configured with more HBM relative to computing power is declining. In the 8xH100 (80GB) node, HBM accounts for more than 60% of the total BOM.

3. The economic value of caching strategies is highlighted again

Epoch AI's data hints at a reverse opportunity: Technology stacks that can reduce HBM request volume will gain significant cost-effective advantages. DeepSeek's prefix-cache mechanism can maintain a cache hit rate of 90%+ in long sessions, and the cache token cost of an API call is only 1/5 of the original. As the cost ratio of HBM continues to rise, this type of cache-first architecture design will become the core competitiveness.

Adaptation suggestions

Suggestions for the team

• Prioritize the use of APIs that support prefix-cache: Models such as DeepSeek that provide explicit cache discounts will directly benefit from the relief of HBM cost pressure
• Focus on the memory efficiency of local inference: If you deploy your own model, consider memory-optimized inference frameworks such as Flash Attention and PagedAttention - they directly reduce HBM occupancy, thus reducing the hardware cost of a single inference.
• Broken down API costs into "cached vs non-cached": Unbundling the cache hit rate metric in cost monitoring of AI applications, a key cost optimization item in the future

Example: Budget Estimate

# API HBM
monthly_api_calls = 500000
avg_tokens_per_call = 4000
cost_per_million_tokens = 0.50

#
no_cache_cost = (monthly_api_calls * avg_tokens_per_call / 1_000_000) * cost_per_million_tokens
print(f": ${no_cache_cost:.2f}")

# （ 90% ， 1/5）
cached_tokens = monthly_api_calls * avg_tokens_per_call * 0.9
uncached_tokens = monthly_api_calls * avg_tokens_per_call * 0.1
cache_cost = (cached_tokens / 1_000_000) * (cost_per_million_tokens / 5)
cache_cost += (uncached_tokens / 1_000_000) * cost_per_million_tokens
print(f": ${cache_cost:.2f}")

• Epoch AI: Memory has grown to nearly two-thirds of AI chip component costs（）
• Microsoft FY2026 $190B capex
• DeepSeek prefix-cache 80%

DeepSeek、Nvidia、OpenAI、Claude, they appear naturally in the text, and the platform side will automatically match the maintained tool library and generate floating cards.

Internal link guidance

• Want to learn how to save AI costs with caching strategies? See: DeepSeek Reasonix in action: Build an AI programming agent at zero cost (30-minute tutorial)
• Real case: How AI Agent achieves profitability through cost optimization: He Built an AI Automation Stack with Claude + n8n — $4K to $12K/mo in 6 Months