LLM Inference Speed and the Memory Hierarchy

A large model that emits a few dozen tokens per second is usually not bottlenecked by compute, but by the speed of pulling weights out of memory. On the very same H100, running a prompt of several thousand tokens drives the compute units almost to saturation — yet the moment it enters token-by-token generation, utilization falls to single-digit percentages. What holds it back is not floating-point throughput; it is memory bandwidth. [2]

To make sense of this, we first have to split LLM inference into two phases. The first is prefill: the entire input prompt is fed into the model at once, computing the attention key/value for every subsequent token and writing them into the KV cache. This step is a large matrix multiply (GEMM) — thousands of tokens march through the network in parallel, arithmetic intensity is high, and it is compute bound. The second phase is decoding (token-by-token generation): the model autoregressively produces one token at a time, feeds it back as input to produce the next, and loops until it stops. This is where the problem lives — to generate each token, the entire model’s weights (plus an ever-growing KV cache) must be read from memory in full, yet only a minute amount of computation is done per byte read. The compute units spend most of their time waiting for data, so decoding is memory bandwidth bound. [1][2]

This explains why the past decade of seemingly scattered improvements on the GPU memory side all point at the same goal — feeding the bandwidth. VRAM climbed from HBM2 up through HBM3e and HBM4, and per-card bandwidth rose from the P100’s ~0.7 TB/s to the B200’s 8 TB/s; on-die SRAM grew ever larger, with L2 going from 4 MB to over a hundred MB and per-SM shared memory from 64 KB to 228 KB; the memory bus grew ever wider, with a single HBM stack reaching 1024 or even 2048 bits. Behind these numbers sit three storage technologies (SRAM, GDDR, HBM), a handful of oligopoly suppliers, and a packaging line being crushed by AI demand.

This article follows that thread: first why decoding is memory bound, then the GPU memory hierarchy and its generational evolution, then the fabrication and cost, and finally where the supply-chain bottleneck lies.

Decoding and Bandwidth

Autoregression — every token reads the whole model once

A Transformer generates text autoregressively — the output of the $t$-th token depends on all preceding tokens, so the model must compute the $t$-th token before it can feed it in to compute the $t+1$-th. This chained dependency cannot be parallelized along the time axis; it can only proceed one token at a time.

The crux is this: generating each token requires the entire set of model weights in the forward pass. A dense model with a given parameter count must read that many weights out of memory to perform one round of multiply-accumulate. If weights are stored in FP16 (2 bytes per parameter), then generating each token moves at least “parameter count × 2” bytes from memory to the compute units. On top of that, the KV cache must be read — every previous token left a key/value in each layer, so the longer the sequence, the larger this read. [2]

And the compute? The core operation of decoding is “matrix × vector” (one token is one vector), not the “matrix × matrix” of prefill. A matrix-vector product does exactly one multiply-accumulate per weight — a weight is read in, used once, and discarded, with almost no data reuse.

Arithmetic intensity and the roofline

The standard tool for deciding whether a computation is “starved for compute” or “starved for bandwidth” is the roofline model, whose central metric is arithmetic intensity — how many floating-point operations are performed for each byte moved from memory:

$I = \frac{\text{FLOPs}}{\text{Bytes moved}}$

Decoding’s arithmetic intensity is extremely low. To generate a token, reading one FP16 weight (2 bytes) does roughly 2 floating-point operations (one multiply, one add), working out to $I \approx 1$ FLOP/byte in order of magnitude. [2] The “ridge point” of an H100 — peak compute divided by peak bandwidth — is about $990 \text{ TFLOPS} \div 3.35 \text{ TB/s} \approx 295$ FLOP/byte. Decoding’s arithmetic intensity is two orders of magnitude below the ridge point, sitting firmly on the bandwidth-bound side of the roofline: performance is set by the memory-bandwidth slope, and the peak-compute ceiling never comes into play. [1]

Prefill is the exact opposite. Processing hundreds or thousands of tokens at once, a single read of the weights serves an entire batch of tokens, the operation becomes a “matrix × matrix” GEMM, each weight is reused hundreds or thousands of times, arithmetic intensity easily reaches several hundred FLOP/byte, crosses the ridge point, and enters the compute-bound region. So for the same model, prefill competes on compute and decoding on bandwidth — this is the first dividing line for understanding LLM inference performance.

A rule-of-thumb formula

Compressing the reasoning above into a formula for estimating generation speed:

$\text{tokens/s} \approx \frac{\text{memory bandwidth (bytes/s)}}{\text{bytes read per generated token}}$

Take Llama-70B in FP16: weights are about $70 \times 10^9 \times 2 = 140$ GB. A single request (batch size 1) on one H100 (3.35 TB/s) has a theoretical ceiling of about $3350 \div 140 \approx 24$ tokens/s; on a B200 (8 TB/s) about $8000 \div 140 \approx 57$ tokens/s. [2] In practice — KV-cache reads, kernel-launch overhead, bandwidth-utilization discounts — it lands somewhat below this ceiling, but the order of magnitude and the ceiling it sets hold. This is exactly why moving the same model to a higher-bandwidth card makes single-stream generation almost linearly faster.

Incidentally, batching is the main way to sidestep the bandwidth wall — stitching many requests into one batch lets a single read of the weights serve dozens or hundreds of tokens at once, so arithmetic intensity rises with batch size and decoding gradually slides from bandwidth-bound toward compute-bound. [2] But this improves total throughput; the latency of a single request (the generation speed each user sees) is still bandwidth-constrained. To make a single stream faster there is only one road — higher memory bandwidth. And that brings the topic to hardware.

The GPU Architecture and Memory Hierarchy

SM, CUDA core, and Tensor Core

The bulk of the compute in a modern datacenter GPU comes from dozens to over a hundred SMs (Streaming Multiprocessors). Each SM contains a set of CUDA cores (for general FP32 / INT32 scalar work) and several Tensor Cores (units dedicated to matrix multiply-accumulate (MMA), each instruction swallowing a small matrix tile — the workhorse of deep learning). The H100 has 132 SMs; the B200 reaches 148 SMs by stitching two dies together. [3][5]

The GPU hides latency with massive threading: each SM keeps dozens of warps (groups of 32 threads) resident at once, and when one warp stalls waiting on memory, the scheduler immediately switches to another ready warp so the compute units never idle. For this mechanism to work, a clearly tiered memory hierarchy must feed data behind the scenes.

Six tiers, from register file to host memory

The GPU memory hierarchy is structurally the same as a CPU’s, but with its own trade-offs. Counting outward from closest to the compute units:

• Register file — hugging the CUDA cores / Tensor Cores, sub-nanosecond latency. The GPU register file is made enormous (256 KB per SM on the H100, 65,536 × 32-bit entries) because it must keep dozens of warps and thousands of threads live at once; register usage directly determines how many threads an SM can keep resident, i.e. its occupancy.
• Shared memory / L1 — one block per SM, a few hundred KB, single-digit-nanosecond latency. It shares the same SRAM as the L1 cache and can be partitioned by the programmer as the kernel requires. Shared memory is an on-chip scratchpad that the programmer manages explicitly (CPU caches are transparent to software — a key GPU difference); operators like FlashAttention rely on it to keep the attention matrix on-chip and avoid repeated round-trips to VRAM.
• L2 cache — shared across the whole GPU, tens to over a hundred MB, low tens of nanoseconds. It is the gatekeeper to VRAM — every hit saves one HBM access, especially valuable for bandwidth-sensitive workloads like decoding. [6]
• Global memory / VRAM — i.e. HBM (datacenter) or GDDR (consumer), tens to hundreds of GB, hundreds of nanoseconds of latency, several TB/s of bandwidth. Model weights and the KV cache live here, and every decoding step reads the weights out of this tier in full. The bandwidth of this tier is the ceiling on generation speed.
• Host memory — DDR / LPDDR on the CPU side connected over PCIe / NVLink-C2C, largest capacity but far worse latency and bandwidth; only what does not fit in VRAM (extra-large models, extra-long contexts) spills here.

The tiers differ by several orders of magnitude in capacity, bandwidth, and latency — the higher, the faster, smaller, and more expensive; the lower, the larger, slower, and cheaper. That is the entire tension of the memory pyramid. The whole hierarchy exists for one purpose: to keep the Tensor Cores from idling while waiting for data.

From Pascal to Blackwell — stacking bandwidth generation by generation

Over the past decade, nearly every generation of NVIDIA datacenter GPU has done the same thing on the memory side — swap in faster VRAM, stack more on-die SRAM, widen the bus. Generation by generation: [3][4][5]

• Pascal (2016, P100) — the first GPU to use HBM2: 4-layer stack, 16 GB, ~732 GB/s, decisively outrunning the GDDR5 bandwidth of its day. L2 was only 4 MB, per-SM shared memory 64 KB, and it had no Tensor Cores — deep learning ran on brute-force FP16 in the CUDA cores.
• Volta (2017, V100) — HBM2 raised to 900 GB/s, 16 / 32 GB, L2 up to 6 MB. First to introduce Tensor Cores (1st gen), purpose-built to accelerate FP16 matrix multiply — the turning point where the GPU shifted from “general parallel processor” to “deep-learning accelerator.” Per-SM L1 + shared memory merged into 128 KB, with shared memory configurable up to 96 KB.
• Turing (2018, T4 / RTX 20 series) — the datacenter T4 switched to GDDR6 (320 GB/s), taking the low-power inference route; 2nd-gen Tensor Cores added INT8 / INT4 support, paving the way for inference quantization.
• Ampere (2020, A100) — HBM2e, 40 / 80 GB, 1.55 / 2.0 TB/s, with L2 ballooning to 40 MB (ten times Pascal). 3rd-gen Tensor Cores introduced TF32, BF16, and structured sparsity. Per-SM shared memory reached 164 KB. The A100 was the workhorse of the pre-ChatGPT wave of large-model training.
• Hopper (2022, H100 / H200) — switched to HBM3: the H100 at 80 GB, 3.35 TB/s; the 2023 H200 upgraded to HBM3e at 141 GB, 4.8 TB/s (stacking capacity and bandwidth expressly for memory-bound inference). L2 up to 50 MB, per-SM shared memory 228 KB. 4th-gen Tensor Cores + the Transformer Engine natively support FP8 — compressing a weight from FP16’s 2 bytes to 1 byte, which directly halves “bytes read per token” and doubles generation speed. [3]
• Blackwell (2024, B200 / GB200) — HBM3e, 192 GB, 8 TB/s. The B200 uses NV-HBI (a 10 TB/s die-to-die link) to stitch two dies into one logical GPU: 208 billion transistors, 148 SMs, ~126 MB of L2 combined. 5th-gen Tensor Cores introduce FP4, halving bytes-per-token again. GB200 = 2 B200 + 1 Grace CPU, using NVLink-C2C to let the GPU access the CPU-side LPDDR directly at high speed. [5]
• Blackwell Ultra (2025, B300) — HBM3e stacked up to 288 GB, expanding capacity further for extra-long contexts and large-batch inference.

Rolled up into a table (datacenter flagship SKUs, SXM variants):

One clear main line: bandwidth grew about 11× from Pascal to Blackwell, exactly tracking an ~11× rise in the decoding ceiling. And each Tensor Core generation introduced lower precision (FP16 → FP8 → FP4), doubling generation speed again from the “fewer bytes read per token” side — hardware presses on this memory wall from both the bandwidth and the data-volume ends at once.

Storage Fabrication — SRAM, GDDR, HBM

The three electrical storage tiers of the pyramid — on-die SRAM, the GDDR of consumer cards, the HBM of the datacenter — determine the entire tension between bandwidth, capacity, and cost. Their cells work on different principles, and their process difficulty and cost differ by orders of magnitude.

SRAM — on-die register file / cache / shared memory

The register file, L1 / shared memory, and L2 are all SRAM (Static RAM), etched directly onto the GPU die. Its heart is the 6T cell — six transistors forming a cross-coupled bistable loop that, once powered, keeps latching a 0 or a 1 with no refresh required (the origin of “static”). A read senses the node voltage directly, so latency reaches sub-nanosecond; because it shares the same process as logic circuits, it can be etched onto the same silicon as the compute units. [11]

The price is low density, large area, and an extremely high cost per unit capacity. Each bit needs six transistors, and at advanced nodes the SRAM cell has essentially stopped shrinking — TSMC’s N5 6T bit cell is about 0.021 μm², while N3 only shrank to about 0.0199 μm² (a mere 5% smaller). [11] This is why, however much a GPU wants to grow its L2, it can only reach the low hundreds of MB before the silicon area and cost become unacceptable. Its advantage is entirely in bandwidth and latency — the aggregate bandwidth of on-die SRAM is tens of times that of HBM, and the reason FlashAttention is fast is essentially that it keeps hot data on this tier rather than dropping it to HBM.

GDDR — consumer graphics-card VRAM

GDDR is the VRAM of consumer graphics cards — discrete DRAM chips soldered onto the PCB around the GPU. Its cell is 1T1C (one transistor + one capacitor), storing a 0 / 1 as charge on the capacitor; the charge leaks and must be refreshed every few milliseconds, hence “dynamic” — and hence higher latency than SRAM, but a far smaller cell and a far cheaper cost per bit. [13]

GDDR’s generational advance has in recent years relied almost entirely on signal modulation to squeeze the bandwidth limit out of the same PCB traces:

• GDDR6 (2018) — NRZ two-level (high / low = 1 bit/symbol), 14 to 18 Gbps/pin.
• GDDR6X (2020, Micron-exclusive) — switched to PAM4 four-level (2 bit/symbol): frequency unchanged, but each symbol carries one more bit, reaching 19 to 24 Gbps; the price is that the spacing between levels is halved, SNR degrades, and power rises.
• GDDR7 (2024–2025, Blackwell RTX 50 series) — JEDEC pivoted to PAM3 three-level (~1.5 bit/symbol), 28 to 32+ Gbps. It looks like a step back from PAM4, but the three-level spacing is 50% wider, so SNR and bit-error rate are actually better and power is lower at equal bandwidth. [7]

An RTX 5090 uses GDDR7 on a 512-bit bus, about 1.8 TB/s. GDDR costs 4 to 10 times less per unit of bandwidth than HBM — gaming cards can afford it, so consumer GPUs use GDDR universally; but it cannot reach HBM’s ultra-wide bus and ultra-large capacity, so it cannot carry the datacenter’s large models. That is where HBM steps in.

HBM — stacking + silicon interposer, datacenter VRAM

HBM (High Bandwidth Memory) is the most process-intensive branch of the DRAM family. Its storage cell is fundamentally no different from ordinary DRAM — the premium is in “how it’s stacked”: [8]

• multiple DRAM dies (4 to 16 layers) are stacked vertically;
• through-silicon vias (TSVs) are drilled through the chips to form vertical inter-layer connections;
• the whole stack is co-packaged in close proximity with the GPU through a silicon interposer;
• the interface bus is extremely wide — 1024 bits per stack (doubling to 2048 bits from HBM4), far exceeding GDDR’s 32 bits per chip.

Per-stack bandwidth has grown about 16× in ten years. Laying out the main generations (approximate values): [7][8]

Note that HBM4’s pin rate actually drops from 9.2 Gbps to 8 Gbps — it pushes bandwidth up by doubling the bus width to 2048 bit/stack rather than raising frequency further. HBM4 also has a structural change: the base die (the bottom logic layer of the stack) moves from a DRAM process to a logic process (TSMC N5 / N3), so it can integrate a custom controller — marking HBM’s shift from “commodity part” to “a customized component co-designed three ways by GPU vendor, foundry, and memory maker.” [8]

Cost, in order of magnitude (per GB, ballpark):

Why is HBM so expensive? Three accounts: stacking yield (a single open circuit among the thousands of TSVs across 16 layers scraps the whole stack), packaging cost (a CoWoS silicon interposer — a 12-inch wafer yields only a few dozen large interposers), and doubled material (for the same number of bits, HBM uses about 3× the wafer of standard DRAM). These three push the topic to the supply chain — because they decide where capacity is stuck.

Suppliers and Capacity

HBM — three-way oligopoly, SK Hynix in front

HBM is a highly concentrated market — only three companies worldwide can mass-produce it: SK Hynix, Samsung, and Micron. They are also the three DRAM giants (together controlling about 95% of the DRAM market). [9]

HBM market share in 2025 (by revenue, quarterly figures fluctuate):

SK Hynix’s lead comes largely from its choice of packaging process — it uses MR-MUF (Mass Reflow Molded Underfill — apply underfill first, then reflow the whole assembly), which dissipates heat well and yields high, and won NVIDIA’s big orders first; Samsung’s NCF (film bonding layer by layer) long failed to clear the HBM3e yield bar, losing most of the orders, so it pins its comeback on HBM4. The 2026 battle is precisely over HBM4. [9]

Packaging — CoWoS is the deepest physical bottleneck

Once HBM is made it still has to be packaged together with the GPU, and this step is stuck on TSMC’s CoWoS (Chip-on-Wafer-on-Substrate) — placing the GPU die and the HBM stacks together on a silicon interposer for co-packaging. CoWoS has three variants: CoWoS-S (silicon interposer, mainstream, used by H100 / B200), CoWoS-L (LSI bridge, cost-reduced), and CoWoS-R (RDL redistribution, thin-form). [5][10]

CoWoS capacity has long been the physical root cause of NVIDIA high-end card shortages. TSMC’s monthly CoWoS capacity grew from about 13,000 wafers at the end of 2023 to about 35,000 at the end of 2024 and about 75,000 at the end of 2025, targeting about 130,000 by the end of 2026 — roughly a tenfold expansion in three years, yet the CEO still calls capacity “extremely tight, sold out for all of 2026.” TrendForce estimates the supply-demand gap only narrows from about 20% to about 10% by the end of 2026. [10]

The DRAM market — a super-cycle crushed by AI

AI’s appetite for HBM is pushing the entire memory market into a super-cycle. With the same wafer capacity, makers prefer to build the far-higher-margin HBM, so the supply of standard DRAM (DDR5 / LPDDR) is squeezed; add that HBM eats about 3× the wafer per bit, and the same capacity yields fewer total bits. The result is that in 2025–2026 ordinary DRAM prices rose and supply tightened too, spilling over even into server and consumer memory. [9]

All three makers are expanding aggressively in 2026 — SK Hynix and Samsung both announced higher memory capacity to chase AI demand — but building a new fab takes years, so in the near term HBM and advanced packaging remain the two tightest links in the AI compute supply chain. In other words, what limits the scaling of large-model inference right now is often not whether a faster GPU can be designed, but how much HBM can be made and how much CoWoS can be packaged.

Conclusion

Compressing the whole article into one chain:

decoding generates token by token → each token reads the whole model from VRAM once, arithmetic intensity ≈ 1 FLOP/byte → memory bound → generation-speed ceiling ≈ memory bandwidth ÷ bytes read per token → so GPUs keep stacking bandwidth (HBM2 → HBM3e, 0.7 → 8 TB/s), stacking on-die SRAM (L2 4 → 126 MB), and lowering precision to cut bytes read (FP16 → FP8 → FP4) → while the ceiling on bandwidth is itself locked by three process / supply bottlenecks: SRAM that won’t shrink, HBM stacking that is expensive, and CoWoS packaging capacity that is stuck.

So the next time you see “model X runs Y× faster on the H200,” “HBM4 enters mass production,” or “TSMC expands CoWoS again,” each lands on some link of this chain — all of them are, on the same memory wall, pushing again from the bandwidth, the data-volume, or the capacity side. Prefill competes on compute, decoding on bandwidth; and decoding is where users actually feel “fast” or “slow.”

References — Papers · Architecture whitepapers · Supply-chain data

• Williams, Waterman, Patterson. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 2009. dl.acm.org
• Yuan et al. LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv:2402.16363, 2024. arxiv.org
• NVIDIA. NVIDIA Hopper Architecture In-Depth. developer.nvidia.com
• NVIDIA. NVIDIA H100 Tensor Core GPU Architecture Whitepaper. resources.nvidia.com
• NVIDIA. NVIDIA Blackwell Architecture Technical Brief (B200 / GB200). nvidia.com
• Chips and Cheese. Nvidia’s H100: Funny L2, and Tons of Bandwidth. chipsandcheese.com
• JEDEC. GDDR6 / GDDR7 / DDR5 Official Standards. jedec.org
• JEDEC / SK Hynix / Micron. HBM3 / HBM3e / HBM4 (JESD270) Standards and Technical Materials. news.skhynix.com
• TrendForce / IDC. 2025 HBM and DRAM Market Share and Supply-Demand Analysis. trendforce.com
• TSMC. CoWoS Advanced Packaging Capacity and Roadmap. tsmc.com
• Hennessy, Patterson. Computer Architecture: A Quantitative Approach, 6th ed. — Ch. 2 memory hierarchy, appendices on roofline and the GPU memory model.
• Micron. GDDR7 / HBM3e Product Technical Documentation. micron.com
• TechInsights. Reverse-Engineering and Teardown Data Across DRAM / HBM / GPU Generations. techinsights.com