The race for AI infrastructure superiority is shifting its focus from merely stacking computational power to achieving system-level optimization. As AI transitions from model training into more complex realms like inference, autonomous agents, and long-context workflows, processors are no longer the sole constraint.

Network interconnects, memory bandwidth and capacity, along with the efficiency of data transfer between different layers, are now becoming the decisive factors for the next phase of infrastructure upgrades. Insights from a recent AI infrastructure summit indicate that major players are aligning their strategies to address these emerging challenges.

Networking and Memory Emerge as the New Frontier

According to notes from the event, executives from Broadcom, Marvell Technology, Samsung, and Amazon Web Services (AWS) detailed their respective blueprints for networking, storage, and custom silicon. A common theme resonated throughout their presentations: as AI workloads grow in complexity, infrastructure must simultaneously solve for interconnect efficiency, memory capacity, bandwidth, power consumption, and overall cost. Broadcom articulated a clear vision that Ethernet will be the preferred interconnect solution for large-scale AI infrastructure, claiming its architecture can achieve over 30% savings in five-year total cost of ownership (TCO) compared to alternatives.

Samsung unveiled its HBM roadmap, projecting that its next-generation zHBM technology could deliver a 4 to 8 times performance boost per GPU over HBM5 while simultaneously reducing thermal resistance by an impressive 75% to 90%. Marvell highlighted the critical role of memory hierarchy optimization, particularly for handling the explosive growth of KV caches in inference-heavy and long-context scenarios. The company warned that when these caches exceed HBM capacity, systems will be forced to reload data from slower memory layers or even recompute context, leading to higher inference latency and increased power draw.

This dynamic underscores a fundamental shift: future AI infrastructure must not only add compute but also re-architect memory hierarchies and data paths to ensure fluid movement between various computing and storage tiers.

HBM Innovation and Custom Chips for System Efficiency

Broadcom's Vice President of Product Management detailed the company's AI networking portfolio, which includes NICs, switches, and software. The Tomahawk series continues to double bandwidth with each generation, with the latest Tomahawk 6 reaching 102.4 Tbps designed for massive AI clusters. Broadcom posits that enhanced network efficiency directly translates to lower power, cooling, and operational costs, reinforcing its TCO advantage.

Samsung's DRAM Product Planning executive outlined a technological evolution path from generative AI to agentic AI and eventually physical AI. With training demands expected to grow roughly 3-fold and inference needs potentially surging by 100-fold, the importance of memory, storage, power, and thermal management will escalate dramatically. The company detailed its HBM4E achieving 16 TB/s bandwidth per GPU with a 20% improvement in performance-per-watt over HBM4, and HBM5 doubling performance again. Samsung also introduced its Z-NAND solution, which combines SLC NAND with TSV stacking to offer read performance comparable to LPDDR5 DRAM at roughly ten times the bit density, with sampling anticipated in 2028.

AWS is leveraging its custom silicon strategy, having developed a family of chips including Graviton, Inferentia, and Trainium. The company's Senior Vice President emphasized that Trainium's core advantage lies in its "memory bandwidth per unit of cost," a metric that directly impacts the economics of AI training and token efficiency. As AI moves from model training into real-world agents and workflows, the competition's focus is evolving. While raw compute scale remains important, network connectivity, memory bandwidth and capacity, and data transfer efficiency are emerging as the key variables that will dictate both the performance and cost-effectiveness of future AI systems.