Colloquium Details
On the Connection between Deep Neural Networks and Kernel Methods
Speaker: Ronen Basri, Weizmann Institute of Science
Location: 60 Fifth Avenue 150
Date: February 3, 2023, 11 a.m.
Host: Davi Geiger
Synopsis:
Recent theoretical work has shown that under certain conditions, massively overparameterized neural networks are equivalent to kernel regressors with a family of kernels called Neural Tangent Kernels (NTKs). My work in this subject aims to better understand the properties of NTK for various network architectures and relate them to the inductive bias of real neural networks. In particular, I will argue that for input data distributed uniformly on the sphere NTK favors low-frequency predictions over high-frequency ones, potentially explaining why overparameterized networks can generalize even when they perfectly fit their training data. I will further discuss the behavior of NTK when data is distributed nonuniformly and show that NTK (with ReLU activation) is tightly related to the classical Laplace kernel, which has a simple closed-form. Finally, I will discuss our analysis of NTK for convolutional networks, which indicates that these networks are biased toward learning low frequency target functions with any higher frequencies concentrated in local regions. Overall, our results suggest that much insight about neural networks can be obtained from the analysis of NTK.
Speaker Bio:
Ronen Basri received the Ph.D. degree from the Weizmann Institute of Science. He later was a postdoctoral fellow at the Massachusetts Institute of Technology and subsequently joined the Weizmann Institute of Science, where he currently holds the position of Professor and incumbent of the Elaine and Bram Goldsmith Chair of Applied Mathematics. At Weizmann he further served as Dean of Math and Computer Science and as Head for the Department of Computer Science and Applied Mathematics. He recently joined Meta Platforms Inc. as an AI Research Scientist. His research interests include computer vision and machine learning. His work deals with object recognition, shape modeling and reconstruction, lighting analysis, and image segmentation.
Notes:
In-person attendance only available to those with active NYU ID cards.