Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shift

Mona Schirmer, Dan Zhang, Eric Nalisnick
12/13/2023
cs.LGcs.AI

No confident code implementation found for this paper.

Abstract

Knowing if a model will generalize to data 'in the wild' is crucial for safe deployment. To this end, we study model disagreement notions that consider the full predictive distribution - specifically disagreement based on Hellinger distance, Jensen-Shannon and Kullback-Leibler divergence. We find that divergence-based scores provide better test error estimates and detection rates on out-of-distribution data compared to their top-1 counterparts. Experiments involve standard vision and foundation models.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Assessing Generative Models via Precision and Recall

81% match1 repo1132018

Demystifying Disagreement-on-the-Line in High Dimensions

81% match2023

Rejection via Learning Density Ratios

81% match1 repo12024

Bridging Domain Expertise and Generalization for Performance Estimation

80% match2026

Assessing Model Generalization in Vicinity

79% match1 repo62024

Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning

79% match2022

Cite this paper

@article{schirmer2023beyond,
  title  = {Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shift},
  author = {Mona Schirmer and Dan Zhang and Eric Nalisnick},
  year   = {2023},
  eprint = {2312.08033},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.08033v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.08033.svg)](https://codeofpaper.com/paper/2312.08033)

Discussion