C-BRIC e-Workshop - Stability explains double descent and deep learning.

Abstract:

Supervised learning is a paradigm in which natural or artificial networks learn a functional relationship from a set of input-output training examples. A central question for the theory is to determine conditions under which a learning algorithm will be able to predict well on new inputs after training on a finite training set. A milestone in classical learning theory was a characterization of the hypothesis space -- that is the space of functions represented by the networks -- that ensure generalization and consistency for the class of empirical risk minimization (ERM) learning algorithms. The classical theory assumes that the hypothesis space is fixed while the number of training data $n$ increases to infinity. Its basic results are thus in the ``classical'' regime of $n > d$, where $d$ is the number of parameters to be learned. The classical theory, however, cannot deal with what we call here the ``modern'' regime, in which the network remains overparameterized ($n<d$) when $n$ grows, since $d$ is also required to grow. Here we show that optimization of a certain form of stability is sufficient to guarantee predictivity in the classical as well as in the modern regime, thus providing a unifying principle that explains the good predictivity properties of deep networks as well as the double descent curve found recently in several algorithms including kernel machines.

Bio:

Tomaso Poggio is Eugene McDermott Professor in the Department of Brain and Cognitive Sciences and at the Artificial Intelligence Laboratory. He is a founding member of the McGovern Institute, and is also the director of the Center for Brains, Minds, and Machines, a multi-institutional collaboration headquartered at the McGovern Institute. He joined the MIT faculty in 1981, after ten years at the Max Planck Institute for Biology and Cybernetics in Tubingen, Germany. He received a Ph.D. in 1970 from the University of Genoa. Poggio is a Foreign Member of the Italian Academy of Sciences and a Fellow of the American Academy of Arts and Sciences. He was awarded the 2014 Swartz Prize for Theoretical and Computational Neuroscience.