Looking back from today to H2 2025, that period marked a pivotal inflection for the AI hardware supply chain. Before then, 'compute' power, exemplified by NVIDIA GPUs, sat center stage. After that point, 'memory' power — spanning DRAM, NAND, and HDD — took the baton and became the biggest bottleneck across the stack.

As the chart shows, DDR5 prices surged nearly 7–9x within six months after Sep 2025, while NAND prices climbed 4–5x over the same span. In tandem, the share prices of memory leaders Micron and SK Hynix spiked by about 10x at one point in that half-year window.

We at Dolphin Research have finally completed a comprehensive review of this forceful 'memory power' cycle and mapped the outlook ahead as part of a major thematic series. A few days ago, we used SanDisk as a case study for a preliminary look at NAND (see link).

In this piece, we open our discussion on high-performance HBM, which is in even shorter supply. As the first article, we approach it from the ground up and focus on three questions. What triggered the 'sudden' surge in AI memory demand over the past year, and what are LLMs’ core memory requirements?

Why did HBM — High Bandwidth Memory — become the default memory spec for top-tier GPUs, and what technology set it apart? What are the challenges and trade-offs behind HBM’s design, and how could they affect the memory supply chain and vendors?

We lay this foundation to discuss future technology roadmaps, vendor differentiation, and ultimately the investment logic and implications in subsequent pieces. $Micron Tech(MU.US) $SK Hynix(SKHY.US) $Samsung Electronics (SSNGY.US)

Below is the full text:

I. Why has 'memory power' become the biggest bottleneck?

As AI advances, demand for DRAM/NAND has exploded to the point where memory has at times been seen as a bigger bottleneck than compute. Why did AI’s memory demand and performance requirements spike seemingly overnight? Two angles: more capacity and faster data transfer.

1.1 More capacity required

In short, the need for more DRAM capacity stems from rising parameter counts, larger context windows, and Agent-driven workflows. These three factors drive the bulk of the increase.

a. Parameters: First, LLMs need memory to store the full set of model parameters. This requirement scales roughly linearly with total parameters and the bytes per parameter. Flagship models have grown by orders of magnitude in the past 3–4 years, pushing capacity needs higher.

Take the Qwen family: from the first Qwen to Qwen 3.8 Max, total parameters rose from 72 bn to about 2.4 tn. Per model instance, parameter storage rose from ~135 GB to nearly ~2,300 GB — roughly 17x.

(Earlier LLM generations typically used BF16 storage at 2 bytes per parameter; recent models store at FP8 or lower, at ~1 byte per parameter or less.)

b. KV cache: Beyond fixed parameter storage per instance, the bigger swing factor is KV cache during inference output. This requirement depends on context length and the attention architecture’s cache compression efficiency, and does not grow strictly linearly.

From Qwen’s first gen to Qwen 3.8 Max, context length expanded by ~31x, broadly in line with parameter growth. But thanks to cache innovations, the required memory growth has been much smaller.

By our estimates, Qwen 3.8 Max’s max context is ~8x Qwen 2.5, yet cache per session is ~36 GB — only about 12% higher than Qwen 2.5. (Estimates assume using 80% of max context.)

Even with software gains, per-session cache growth lags parameter and context growth. However, as users and sessions surge, aggregate cache demand can still rise by tens of times, even up to 100x.

Assume 32 concurrent sessions (at 80% of max context). Total memory per Qwen 3.8 Max instance (params + KV) still reaches about 2.3 + 1.3 = 3.6 TB.

c. Beyond LLM/Agent needs, AI Agents trigger traditional compute tasks like document editing or Excel operations. These also drive DRAM/NAND demand, though they do not require high-spec products like HBM.

1.2 Faster bandwidth required

Capacity aside, AI computing likely prioritizes data transfer speed even more. Two angles: keep pace with faster compute to avoid memory becoming the system bottleneck, and meet the explosion in cache read/write for larger parameters and longer contexts.

a. Stronger chips need faster memory: Using NVIDIA as an example, from B100 to VR200, compute (FP4 dense) is ~5x higher, so memory bandwidth must scale accordingly. If a chip can process 1 GB/s but memory can only feed 0.5 GB/s, the GPU spends roughly half its time idle, wasting compute.

In practice, from B100 to GB300, five generations of GPUs stayed capped at ~8 TB/s memory bandwidth (HBM3e). The bandwidth-to-compute ratio fell from ~1.1 to ~0.5 (FP4 basis).

VR200 moves to HBM4 with ~22 TB/s bandwidth. But VR200 compute rises ~2–3x as well (FP4), leaving the bandwidth-to-compute ratio at ~0.6. Memory bandwidth remains the relative bottleneck.

b. Quadratic growth in inference workload: During inference, data reads scale roughly with the square of generated tokens, not linearly. A 10x increase in generated context can imply ~100x more data reads.

Each new token requires reading all activated parameters plus the entire prior KV cache. Generating n tokens repeats this n times, forming a classic arithmetic series.

For Qwen 3.8 Max, on top of an existing 100k context, generating another 100k tokens implies ~500 TB of data reads. Yet despite larger parameters, Qwen 3’s read/write needs have fallen vs. prior generations, reflecting software progress.

Two drivers help: architectures have shifted from dense to sparse activation, and attention has moved from full to mixed/linear attention. This reduces reliance on full KV reads by compressing or focusing on the most relevant cache.

But per Jevons paradox, software savings are reabsorbed by scaling models to trillions or even tens of trillions of parameters and longer contexts. So even with software efficiency gains, read/write volume and speed remain formidable constraints.

Our discussion here is per model/session. With AI entering the Agent and coding era, the number of users and sessions is exploding, amplifying total memory demand.

1.3 Training: memory is less of a bottleneck
So far we focused on inference. Training has different memory needs, and overall, memory is less of a bottleneck during training. We summarize briefly.

Training prioritizes capacity over bandwidth. First, unlike inference where parameters are fixed, training must store intermediate states and computations, and it still runs at BF16 or higher precision.

Parameter storage during training can be ~8–16x inference. Even so, capacity is less of a bottleneck because training clusters are dedicated to a single model, leaving ample memory, while inference resources are partitioned across many model instances.

Bandwidth needs are also less severe in training because compute intensity is much higher, keeping GPUs fully utilized. Idle cycles from memory wait states are less common.

II. Why HBM stands out

AI’s memory needs boil down to larger capacity and higher bandwidth. Technically, how does HBM deliver on both and become the memory of choice for high-performance GPUs?

HBM and DDR use similar DRAM dies; the key differences lie in architecture and packaging. HBM’s advantages are primarily structural.

The schematic shows two defining features: DRAM dies are vertically stacked, and HBM is co-packaged with the GPU on a silicon interposer. These jointly enable large capacity and very high bandwidth.

Key components include: (i) DRAM dies, (ii) a base/logic die for management, (iii) TSVs for vertical inter-die connections, and (iv) a silicon interposer that hosts and links HBM with the GPU. We detail each below.

2.1 Stacking boosts capacity

There are two ways to raise total capacity: increase per-die density, or fit more dies in the same footprint. Die density depends on process nodes, but DRAM scaling is constrained by physics.

If geometry shrinks too far, electrons can leak, corrupting data. State-of-the-art DRAM nodes are ~11–12 nm and may only reach ~10 nm by 2030, far behind logic at 2–3 nm.

Unlike logic’s Moore’s Law, DRAM density gains via node shrinks are much harder. That leaves the more straightforward path: pack more dies in limited area via vertical stacking.

This is HBM’s signature — stacking multiple DRAM dies to raise capacity per footprint. Under HBM4, stack height reaches 12-high, about 3x the first gen, with reports suggesting 16-high stacks around 2026–27.

2.2 Scarce area and edge constraints

Why not lay out many DRAM packages flat on the substrate to raise capacity as in traditional designs? Because high-performance GPUs have very limited area for memory placement.

To maximize bandwidth, HBM must sit close to the compute die and be co-packaged on a dedicated silicon interposer. This area is limited and cannot be enlarged easily, restricting available space for components.

On consumer GPUs, the silver region hosts only the compute die, with DRAM chips on the surrounding PCB. On server GPUs, HBM packages sit right next to the GPU core within the silver region, where area is tight.

HBM stacks appear only above and below the GPU; the left and right sides are left open. That is because I/O must be routed around the four edges of the compute die.

Those edges also need lanes for die-to-die and external links, further squeezing memory placement. As a result, horizontal area is constrained and vertical stacking is the practical path.

III. HBM’s technical challenges

HBM delivers compelling benefits, but not without cost. Where are the main technical challenges?

3.1 Challenges in stacking

a. Through-silicon vias (TSVs)

First, how do vertically stacked dies communicate and receive power? Enter TSVs, a core enabler in stacked memory.

TSVs are vertical holes etched through silicon and filled with copper to carry signals and power. Each die uses micro-bumps aligned to TSV pads, with controlled spacing for insulation and thermal paths.

TSV fabrication is not uniquely difficult — DRAM vendors can make them — but extra steps raise cost and lower yields. Design choices for TSV layout, power delivery, and thermals contribute to vendor differentiation.

b. Divergent bonding/encapsulation flows

TSVs and micro-bumps enable vertical interconnects, but robust bonding and encapsulation are needed for reliability. SK Hynix uses MR-MUF (mass reflow + mold underfill), while Samsung and Micron use TC-NCF (thermo-compression + non-conductive film).

In TC-NCF: a solid NCF layer is applied to the lower die, then the upper die is aligned and pressed/heated so micro-bumps pierce the film and bond. The film melts, fills gaps, insulates, and solidifies; the process repeats layer by layer.

In MR-MUF: all dies are aligned at once and temporarily fixed with flux, then the stack goes through a reflow oven to solder all joints simultaneously. A liquid resin (EMC) then fills all gaps to encapsulate the stack, followed by cure and planarization.

MR-MUF offers higher throughput, better yields, and improved thermals, supporting higher GPM — a key advantage for SK Hynix. Layer-by-layer TC risks wafer warpage, die cracking, or bump damage, reducing yields.

Why not copy MR? Hynix holds patents and co-developed the resin with upstream suppliers, limiting competitors; Samsung and Micron have sunk R&D and capex in TC, making switching costly.

Rather than replicate MR, Samsung and Micron aim to leapfrog with hybrid bonding next. Bonding technology choices are another axis of vendor differentiation, which we will revisit.

3.2 Co-packaging

To maximize bandwidth, HBM must be co-packaged with the GPU, which also limits memory placement and adds cost/complexity. Why is this necessary?

Memory bandwidth equals frequency times bus width. Bus width counts how many parallel lanes connect memory and compute, while frequency measures per-lane speed.

Compared with GDDR, HBM’s bandwidth edge comes from massive bus width — 1–2k bits vs. 32 bits for GDDR. On a per-lane basis, HBM’s frequency is actually lower.

Routing thousands of lanes between memory and compute in a tiny area, with minimal crosstalk, demands ultra-fine pitch. PCB etching cannot meet this; only semiconductor processes can.

PCB trace widths are typically ~50 µm and even visible to the naked eye. A silicon interposer can print ~1 µm lines, which is why a dedicated interposer is needed to host and link HBM and the GPU.

3.3 Customized base dies

Similar to the interposer, the base/logic die at the bottom of the stack in HBM4 and beyond increasingly requires external foundry support and co-design. The base die links the DRAM stack above and the interposer below, bridging to the XPU for data and power.

In HBM3e and prior, the base die was simple: PHY for XPU I/O, TSV interfaces to the DRAM stack, and Direct Access (DA) for pre-package testing. At ~20 nm-class, DRAM vendors could design and manufacture it in-house.

With HBM4, base-die nodes moved to ~12 nm, and some flagship variants will adopt 2/3 nm at foundries like TSMC. Samsung can self-fab due to its logic processes.

Why the shrink? HBM4 doubles bus width (1k → 2k), demanding finer PHY, and — more importantly — moves memory controllers and address mapping from the XPU onto the base die.

This places control logic closer to the DRAM to cut command/data hops and frees precious XPU die area. Hence the need for advanced nodes.

Another shift is from general-purpose to customer-specific base dies. NVIDIA’s NVHBM replaces JEDEC PHY with a proprietary die-to-die interface, claiming +30% bandwidth, -15% power, and -25% area for compute.

Other players like Marvell and Rambus have announced their own custom base-die solutions. Customization is becoming the norm.

3.4 Memory no longer belongs solely to 'memory vendors'

Co-packaging and advanced, customized base dies shift industry dynamics. HBM output is no longer determined solely by DRAM vendors’ internal capacity.

It also depends on access to foundry capacity for base dies and advanced packaging (notably TSMC’s CoWoS). High-performance memory is now a shared product across DRAM vendors, packagers, and end customers.

HBM specs and designs are increasingly co-developed with downstream customers, pushing the industry toward customization over standardization. Deep alignment with anchor customers becomes existential; losing one can strand bespoke designs.

For SK Hynix’s HBM4, the DRAM stack is built in-house, but the base die is co-designed with NVIDIA and fabbed by TSMC, with final packaging also at TSMC. This reduces DRAM vendors’ autonomy and may compress their share of the value pool.

That said, AI-driven growth is expanding the total value pie for HBM-class memory, allowing all participants to benefit for now. The industry is in a 'grow the pie' phase.

IV. Closing

From both demand and technology angles, two factors define memory: capacity/density and bandwidth. Today’s differentiation largely stems from stacking and packaging approaches.

Next, we will discuss how future DRAM raises capacity and bandwidth while balancing cost, and what those changes mean for the industry and its players. Stay tuned.

<End>

Risk disclosure and disclaimer:Dolphin Research Disclaimer and General Disclosure