Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learning

Yingxiao Huo, Satya Prakash Dash, Radu Stoican, Samuel Kaski, Mingfei Sun
1/26/2026
cs.LGcs.AIstat.ML

No official code found. Best implementation: agent-lab/sherman-morrison-actor-critic (0 stars, MIT, last commit 2026-02-06).

Abstract

Natural gradients have long been studied in deep reinforcement learning due to their fast convergence properties and covariant weight updates. However, computing natural gradients requires inversion of the Fisher Information Matrix (FIM) at each iteration, which is computationally prohibitive in nature. In this paper, we present an efficient and scalable natural policy optimization technique that leverages a rank-1 approximation to full inverse-FIM. We theoretically show that under certain conditions, a rank-1 approximation to inverse-FIM converges faster than policy gradients and, under some conditions, enjoys the same sample complexity as stochastic policy gradient methods. We benchmark our method on a diverse set of environments and show that it achieves superior performance to standard actor-critic and trust-region baselines.

View on arXivView PDF

Code Implementations(1)

agent-lab/sherman-morrison-actor-criticCommunity70%
MIT

Code base for paper "Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learning"

00Shell, PythonFeb 6, 20268 months agoMIT

Similar Papers

Linear Function Approximation as a Computationally Efficient Method to Solve Classical Reinforcement Learning Challenges

86% match2024

Matrix Low-Rank Approximation For Policy Gradient Methods

85% match2024

An Improved Empirical Fisher Approximation for Natural Gradient Descent

84% match1 repo2024

Reconstructing Deep Neural Networks: Unleashing the Optimization Potential of Natural Gradient Descent

83% match1 repo32024

Natural Gradient Methods: Perspectives, Efficient-Scalable Approximations, and Analysis

83% match2023

A Novel Framework for Policy Mirror Descent with General Parametrization and Linear Convergence

83% match2026

Cite this paper

@article{huo2026rank,
  title  = {Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learning},
  author = {Yingxiao Huo and Satya Prakash Dash and Radu Stoican and Samuel Kaski and Mingfei Sun},
  year   = {2026},
  eprint = {2601.18626},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2601.18626v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2601.18626.svg)](https://codeofpaper.com/paper/2601.18626)

Discussion