An Analytic Solution to Covariance Propagation in Neural Networks

Oren Wright, Yorie Nakahira, José M. F. Moura
3/24/2024
cs.LGcs.AIstat.ML

No official code found. Best implementation: omwright/cov-prop-nn (5 stars, MIT, last commit 2024-04-30).

Abstract

Uncertainty quantification of neural networks is critical to measuring the reliability and robustness of deep learning systems. However, this often involves costly or inaccurate sampling methods and approximations. This paper presents a sample-free moment propagation technique that propagates mean vectors and covariance matrices across a network to accurately characterize the input-output distributions of neural networks. A key enabler of our technique is an analytic solution for the covariance of random variables passed through nonlinear activation functions, such as Heaviside, ReLU, and GELU. The wide applicability and merits of the proposed technique are shown in experiments analyzing the input-output distributions of trained neural networks and training Bayesian neural networks.

View on arXivView PDF

Code Implementations(1)

omwright/cov-prop-nnCommunity100%
MIT

Analysis and synthesis experiments for covariance propagation in neural networks

50PythonMar 8, 20242 years agoMIT

Similar Papers

Probabilistic computation and uncertainty quantification with emerging covariance

85% match1 repo62023

Uncertainty Quantification via Stable Distribution Propagation

85% match2026

Exact closed-form Gaussian moments of residual layers

82% match2026

Bayesian neural networks with interpretable priors from Mercer kernels

82% match2025

Uncertainty Propagation Networks for Neural Ordinary Differential Equations

81% match2025

Post-Hoc Uncertainty Quantification in Pre-Trained Neural Networks via Activation-Level Gaussian Processes

81% match2025

Cite this paper

@article{wright2024analytic,
  title  = {An Analytic Solution to Covariance Propagation in Neural Networks},
  author = {Oren Wright and Yorie Nakahira and José M. F. Moura},
  year   = {2024},
  eprint = {2403.16163},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2403.16163v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2403.16163.svg)](https://codeofpaper.com/paper/2403.16163)

Discussion