The memory wall is the long-standing observation that processors get faster more quickly than memory can feed them. The idea predates AI accelerators, and it was one of the problems HBM was designed to address when AMD and SK hynix began working on stacked memory in the late 2000s. [1]

For AI hardware, the gap was quantified by Amir Gholami and colleagues in AI and Memory Wall, a paper published in IEEE Micro in 2024. Looking across about 20 years of server hardware, they found that peak compute grew about 3.0 times every two years, while DRAM bandwidth grew about 1.6 times and interconnect bandwidth about 1.4 times over the same interval. They argue that memory bandwidth can become the dominant bottleneck for decoder models, especially when serving them. [2]

A useful way to reason about the problem is the roofline model, proposed by Samuel Williams, Andrew Waterman and David Patterson at Berkeley in 2008. It plots attainable performance against a program's arithmetic, or operational, intensity: the number of floating-point operations it performs for each byte it moves from memory. Below a certain intensity, performance is capped by memory bandwidth; above it, by peak compute. [3]

The H100 figures used in the example come from NVIDIA's own specification table, which notes that the headline tensor numbers assume structured sparsity. [4]

Serving a large language model has two phases. In prefill, the model processes the whole prompt at once, which is dense matrix work and makes good use of compute. In decode, it generates one token at a time, and each step has to read the model's weights and the cached attention state from memory while doing comparatively little arithmetic. The researchers behind the Splitwise system found that the token-generation phase underuses a GPU's compute resources, which is consistent with it being limited by memory. [5][2]

Transformers keep the keys and values computed for earlier tokens so they do not have to recompute them at every step. This key-value, or KV, cache grows with every token of context and every concurrent request. The authors of vLLM, presented at SOSP 2023, described the KV cache for each request as large and dynamically growing and shrinking, and showed that managing it in fixed-size pages, like virtual memory, cut wasted memory to near zero and raised serving throughput two to four times compared with earlier systems. [6]

Model designers have responded by reducing the number of key-value heads. Multi-query attention uses a single key-value head, which speeds up decoding but can reduce quality. Grouped-query attention, published at EMNLP 2023, uses an intermediate number of key-value heads and, according to its authors, achieves quality close to standard multi-head attention at a speed comparable to multi-query attention. [7]

Accelerator makers have spent much of each new generation's budget on memory. NVIDIA's A100 80GB offered about 2 TB/s; the H100 SXM 3.35 TB/s; the H200 4.8 TB/s with 141 GB; and each GPU in a DGX B200 has 8 TB/s. NVIDIA's Rubin GPU, ramping into production as of September 2026, lists 288 GB of HBM4 and 19.2 TB/s. [8][4][9][10][11]

Even that growth has not caught up with compute. Using the AI and Memory Wall paper's rates, compute roughly tripling every two years against bandwidth growing about 1.6 times, the ratio of FLOPS to bytes per second keeps rising, so the arithmetic intensity a kernel needs to stay compute-bound rises too. More HBM relieves the pressure without removing it.

If bandwidth is the limit, reading fewer bytes is the most direct fix. The GPTQ method, presented at ICLR 2023, quantised weights to 3 or 4 bits with little loss of accuracy, fitted a 175-billion-parameter model on a single GPU for inference, and reported end-to-end speedups of about 3.25 times over FP16 on an A100. Hardware has followed: NVIDIA's NVFP4 format, supported on Blackwell GPUs, cuts the memory footprint by about 3.5 times relative to FP16 and 1.8 times relative to FP8, and NVIDIA reports accuracy losses of one percent or less on the models it tested. [12][13]

Better memory management helps too. PagedAttention, used in vLLM, lets many more requests share a GPU by removing fragmentation in the KV cache, and it allows cached blocks to be shared within and across requests rather than duplicated. [6]

Because prefill and decode stress different resources, researchers have proposed running them on different machines. Splitwise reported 1.4 times higher throughput at 20 percent lower cost, or 2.35 times more throughput for the same cost and power, by sending prompt processing and token generation to separately optimised pools. DistServe, presented at OSDI 2024, placed prefill and decoding on different GPUs and served up to 7.4 times more requests within latency targets than the systems it compared against. [5][14]

These ideas are now in commercial products. NVIDIA's open-source Dynamo inference framework splits prefill and decode across nodes and includes a KV block manager that moves cache data to cheaper tiers of memory and storage. NVIDIA's Rubin CPX, announced in September 2025 for the end of 2026, pairs 128 GB of GDDR7 with compute aimed at the context-processing phase, leaving HBM-equipped Rubin GPUs to handle generation. [15][16]

For anyone renting GPUs for inference, the memory wall changes what to compare. Peak TFLOPS is a poor guide to tokens per second; memory bandwidth, memory capacity and the number formats a GPU supports usually matter more. An older GPU with a lot of HBM can outperform a newer one with less for long-context serving, and quantising a model can be worth more than a hardware upgrade. On Kovara you can compare GPUs by memory bandwidth and capacity alongside hourly price, look up a specific GPU's memory specifications, or ask Kova to estimate which GPU fits your model and context length.