NVIDIA promotes full-stack codesign to ease AI memory constraints

Traders Union
All News
Business & Wealth
Companies
NVIDIA full-stack AI memory

Highlights

NVIDIA said full-stack codesign combining GPUs, CPUs, networking, storage, and software helps overcome AI memory bottlenecks driven by growing inference and retrieval workloads.
The company reported HBM capacity per GPU rose 3% from Blackwell Ultra to Vera Rubin, but agentic throughput per dollar increased up to 30 times on SemiAnalysis AgentX.
NVIDIA emphasized that maximizing tokens per second per watt is key for AI factory operators to enhance revenue and reduce cost per token within fixed power budgets.

Memory strategy across the AI stack

According to NVIDIA Newsroom, the company is addressing AI memory bottlenecks through full-stack codesign, a strategy that combines GPUs, CPUs, networking, storage and software to raise the amount of useful work delivered from each byte of memory.

The company said the approach is becoming more important as inference workloads require more capacity for model weights and KV cache, while retrieval-augmented generation adds further context that must be processed and retained. In an AI factory, high-bandwidth memory, or HBM, remains the fastest and most expensive memory tier, with lower layers such as CPU memory, local SSDs and shared storage trading speed for greater capacity at lower cost.

NVIDIA said simply expanding HBM on each GPU does not solve the problem because bandwidth limits how much data can move in and out at a given time. Once memory bandwidth reaches capacity, added HBM can raise cost and power use without proportionally improving performance, a constraint engineers describe as the memory wall.

The company pointed to several techniques intended to reduce HBM pressure. These include using lower-precision model weights across the NVL72 domain, shifting inactive context to nearby CPU memory or NVIDIA CMX context memory storage, and using NVIDIA Nemotron Retriever models and the cuVS library to reduce the memory burden from retrieval systems.

Efficiency gains and infrastructure implications

NVIDIA said efficiency gains are outpacing raw HBM growth across its platform generations. From NVIDIA Blackwell Ultra to the Vera Rubin platform, HBM capacity per GPU rose by roughly 3%, while agentic throughput per dollar increased by as much as 30 times on SemiAnalysis AgentX.

For AI factory operators, the company said the more important measure is how many useful tokens a system can deliver within a fixed power budget while still meeting response-time targets. Higher tokens per second per watt can improve revenue potential from the same power envelope and lower the cost per token, making memory efficiency a broader operating and economics issue rather than only a hardware specification question.

Our earlier article on the U.S. AI data center buildout explained how the $10 trillion infrastructure boom is running into a reality check, with many projects delayed or canceled and only a minority viewed as truly credible. It highlighted rising financing risks, construction cost overruns, and intensifying power-grid constraints and local opposition—factors that could slow the pace at which new AI computing capacity comes online.

This material may contain third-party opinions, none of the data and information on this webpage constitutes investment advice according to our Disclaimer. While we adhere to strict Editorial Integrity, this post may contain references to products from our partners.