A Simple Algorithm For Scaling Up Kernel Methods

Teng Andrea Xu, Bryan Kelly, Semyon Malamud
1/26/2023
cs.LGmath.OC

No official code found. Best implementation: tengandreaxu/fabr (4 stars, MIT, last commit 2023-07-06).

Abstract

The recent discovery of the equivalence between infinitely wide neural networks (NNs) in the lazy training regime and Neural Tangent Kernels (NTKs) (Jacot et al., 2018) has revived interest in kernel methods. However, conventional wisdom suggests kernel methods are unsuitable for large samples due to their computational complexity and memory requirements. We introduce a novel random feature regression algorithm that allows us (when necessary) to scale to virtually infinite numbers of random features. We illustrate the performance of our method on the CIFAR-10 dataset.

View on arXivView PDF

Code Implementations(1)

tengandreaxu/fabrCommunity100%
MIT
41Shell, PythonJan 26, 20233 years agoMIT

Similar Papers

Kernel Regression with Infinite-Width Neural Networks on Millions of Examples

88% match2023

How Feature Learning Can Improve Neural Scaling Laws

84% match2024

Convolutional Deep Kernel Machines

83% match1 repo32026

Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theories

83% match2024

Random Features for Operator-Valued Kernels: Bridging Kernel Methods and Neural Operators

83% match2026

Scaling and renormalization in high-dimensional regression

83% match1 repo132024

Cite this paper

@article{xu2023simple,
  title  = {A Simple Algorithm For Scaling Up Kernel Methods},
  author = {Teng Andrea Xu and Bryan Kelly and Semyon Malamud},
  year   = {2023},
  eprint = {2301.11414},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2301.11414v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2301.11414.svg)](https://codeofpaper.com/paper/2301.11414)

Discussion