Keynote Speaker: Stefano Soatto, Vice President, AWS and Professor, UCLA
Talk Title: Where LLM Scaling Laws End
Abstract:
I will share the long-term vision driving the development of next-generation foundation models at AWS. Rather than just scaling current models, we start from the defining characteristics of optimal representations of sequential data, which pertain to any downstream task. These were established in the Seventies in the context of Stochastic Realization Theory. When the mechanisms generating data are non-stationary, as in business, finance, climate, and language data, optimal representations require effectively unbounded complexity, unthinkable in the Seventies but not today. So, if we scale computation, where does it end? Solomonoff answered that question in 1964, showing that optimal inference conspicuosly involves no “intelligence”, no “insight”, no “meaning”, just cycling over programs; yet it performs optimal transduction. It does not even involve learning, just test-time computation over unbounded memory. Are scaling laws pointing us in that direction? Is too much of a good thing (scaling) not a good thing? I will discuss how current LLMs are effectively operating in the transductive setting, albeit suboptimally (through “in-context learning”, which notably involves no learning), and have effectively unbounded memory through RAG. I will finally point to analytical tools that can be used to study the controllability, observability, and stability of the resulting systems, and the implications for their use and interaction with humans.
Speaker Bio:
Stefano Soatto is a Professor of Computer Science at the University of California, Los Angeles and a Vice President at Amazon Web Services, where he leads the AI Labs. He received his Ph.D. in Control and Dynamical Systems from the California Institute of Technology in 1996. Prior to joining UCLA he was Associate Professor of Biomedical and Electrical Engineering at Washington University in St. Louis, Assistant Professor of Mathematics at the University of Udine, and Postdoctoral Scholar in Applied Science at Harvard University. Before discovering the joy of engineering at the University of Padova under the guidance of Giorgio Picci, Soatto studied classics, participated in the Certamen Ciceronianum, co-founded the Jazz Fusion quintet Primigenia, skied competitively and rowed single-scull for the Italian National Rowing Team. Many broken bones later, he now considers a daily run around the block an achievement.
Soatto received the Siemens Prize with the Best Paper Award at CVPR in 1998 (with the late Roger Brockett), the Marr Prize at ICCV 1999 (with Jana Kosecka, Yi Ma, and Shankar Sastry), the Best Paper Award at ICRA 2015 (with Konstantine Tsotsos and Joshua Hernandez). He is a Fellow of the IEEE of the ACM.