On Feature Scaling of Recursive Feature Machines

Arunav Gupta, Rohit Mishra, William Luu, Mehdi Bouassami
3/28/2023
cs.LGcs.AI

No official code found. Best implementation: agupta01/ml-theory-capstone (3 stars, last commit 2025-10-22).

Abstract

In this technical report, we explore the behavior of Recursive Feature Machines (RFMs), a type of novel kernel machine that recursively learns features via the average gradient outer product, through a series of experiments on regression datasets. When successively adding random noise features to a dataset, we observe intriguing patterns in the Mean Squared Error (MSE) curves with the test MSE exhibiting a decrease-increase-decrease pattern. This behavior is consistent across different dataset sizes, noise parameters, and target functions. Interestingly, the observed MSE curves show similarities to the "double descent" phenomenon observed in deep neural networks, hinting at new connection between RFMs and neural network behavior. This report lays the groundwork for future research into this peculiar behavior.

View on arXivView PDF

Code Implementations(1)

agupta01/ml-theory-capstoneCommunity100%

Data Science Capstone Project advised by Mikhail Belkin and Parthe Pandit.

31Python, DockerfileOct 8, 202212 months ago

Similar Papers

Linear Recursive Feature Machines provably recover low-rank matrices

84% match1 repo172024

Scaling and renormalization in high-dimensional regression

82% match1 repo132024

Average gradient outer product as a mechanism for deep neural collapse

82% match2024

Average gradient outer product as a mechanism for deep neural collapse

82% match2026

A Simple Algorithm For Scaling Up Kernel Methods

82% match1 repo42023

Emergence in non-neural models: grokking modular arithmetic via average gradient outer product

81% match2024

Cite this paper

@article{gupta2023feature,
  title  = {On Feature Scaling of Recursive Feature Machines},
  author = {Arunav Gupta and Rohit Mishra and William Luu and Mehdi Bouassami},
  year   = {2023},
  eprint = {2303.15745},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.15745v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.15745.svg)](https://codeofpaper.com/paper/2303.15745)

Discussion