The memory wall is becoming a central obstacle to scaling AI infrastructure, and the semiconductor industry is responding by rearchitecting from the ground up. At the second day of the 2026 AI Infrastructure Summit, key memory makers like Micron, Samsung, and SK Hynix joined forces with chip designers such as Broadcom, Marvell, and Intel, as well as cloud giants including OpenAI, AWS, and Google. The discussions zeroed in on how to tackle the memory wall, with broad consensus forming around the disaggregation of memory and storage into distinct, optimized layers.

This shift carries direct implications for semiconductor investment. Efficiency metrics are replacing raw compute scale as the new benchmark for the industry, moving from simple FLOPs to metrics like tokens per watt (tokens/W) and tokens per dollar (tokens/$). At the same time, interconnect architectures are opening up, and demand for custom silicon is rising, which is placing companies like Broadcom, Marvell, and Astera Labs in the spotlight due to their product roadmaps.

Why Memory Is the Core Constraint on AI Expansion

The central takeaway from the summit — echoing themes from August’s Hot Chips conference — is that memory bandwidth and capacity are growing far slower than model sizes. Transformer models are expanding about 240 times every two years, while memory bandwidth and capacity only double in the same period, creating a widening gap. This structural imbalance is making traditional GPU-centric architectures unsustainable, pushing the industry toward more specialized and layered memory solutions.

Qualcomm showcased its "HBC" (High Bandwidth Compute) approach, which stacks LPDDR memory directly on the compute die. This design delivers roughly 200 times the capacity-to-power ratio of SRAM and about 6 times the bandwidth-to-power ratio of HBM. Samsung’s zHBM follows a similar 3D DRAM stacking path.

SK Hynix highlighted three specialized memory tiers: PIM (Processing-In-Memory) for memory-intensive workloads, which offers up to 288 times the capacity per rack compared to SRAM; HBF (High Bandwidth Flash) for long-context scenarios, with roughly 10 times the capacity of HBM; and the SALT-KV software solution, which enables temperature-aware KV cache scheduling across different memory tiers.

Efficiency Takes Over as the New Law of Scaling

A clear signal from the summit is that the era of simply piling on compute is giving way to an efficiency-driven paradigm. According to BofA Securities research, as AI applications evolve toward agentic AI, the industry’s evaluation criteria are shifting from chasing raw FLOPs to focusing on efficiency metrics like tokens/W and tokens/$ . This change is directly fueling the demand for memory diversification and disaggregation, as different workloads require memory tiers with distinct characteristics rather than relying on a one-size-fits-all high-performance solution.

This has a real impact on both datacenter procurement strategies and chip-maker product plans. Flexibility in memory architecture and power efficiency are becoming the new competitive battlegrounds.

Open Interconnects and Customization Move Forward in Tandem

In the networking domain, Ethernet has largely cemented its dominance in scale-out environments, with openness and interoperability being the keys to its success. Broadcom is pushing Ethernet into every layer of the interconnect hierarchy: for scale-out, its Tomahawk 6 (102.4T) is in high-volume production and deployed across hyperscalers; for scale-up, Thor Ultra NIC and ESUN are ready; and for cross-domain connectivity, Jericho 4 handles the load — all built on open architectures without vertical integration.

Meanwhile, demand for customization is heating up. Marvell offers end-to-end solutions spanning custom XPU attach, all-optical interconnect kits, and multi-protocol support (UALink, NVLink Fusion, ESUN). Astera Labs is building on the universal PCIe protocol, deploying specialized chips like Scorpio and Leo to target both interconnect and memory markets.

Ramp Speed and Reliability Are Just as Critical as Chip Design

A consistent message emerged from frontier labs and datacentre operators at the summit: the speed of rack-level ramp-up and system reliability now matter just as much as the chip design itself. AWS has compressed the traditional 6-to-9-month post-silicon testing and stabilization cycle by aligning test capacity with installation scale and co-locating manufacturing with deployment sites, significantly shortening the time between a chip’s release and its actual deployment in datacenters.

Software co-design is also being highlighted by hyperscalers as a key accelerator for deployment. Native PyTorch support and Hugging Face portability with minimal code changes are seen as critical ways to shorten the journey from chip to application. The BofA Securities research team believes that an annual or faster product iteration cadence is vital for token economics and market competitiveness. Achieving this pace, however, no longer depends purely on chip design skills, but on system-level engineering integration and tight hardware-software collaboration.