Memory optimizations in Machine Learning

The evolution of AI has largely been shaped by advancements in compute power. However, an equally critical factor—memory—has emerged as a defining bottleneck for the next generation of AI infrastructure. While GPUs and TPUs have seen exponential improvements in FLOPS, memory bandwidth and capacity have struggled to keep pace. Today, training and inference at scale are constrained as much by memory limitations as by compute. The financial implications are staggering: High-Bandwidth Memory (HBM) now costs nearly as much as compute, and memory bandwidth is one of the leading constraints in large-scale AI deployments. The infrastructure of tomorrow must be designed with memory as a first-class consideration. This keynote explores the increasing role of memory in AI workloads, real-world examples of memory bottlenecks, and strategies for designing AI infrastructure that balances compute and memory effectively.