Six Lectures on Linearized Neural Networks
Theodor Misiakiewicz, Andrea Montanari
8/25/2023
stat.MLcs.LGmath.ST
No confident code implementation found for this paper.
Abstract
In these six lectures, we examine what can be learnt about the behavior of multi-layer neural networks from the analysis of linear models. We first recall the correspondence between neural networks and linear models via the so-called lazy regime. We then review four models for linearized neural networks: linear regression with concentrated features, kernel ridge regression, random feature model and neural tangent model. Finally, we highlight the limitations of the linear theory and discuss how other approaches can overcome them.
