Informative Features for Model Comparison

Wittawat Jitkrittum, Heishiro Kanagawa, Patsorn Sangkloy, James Hays, Bernhard Schölkopf +1 more
10/27/2018
stat.MLcs.LG

Official code: wittawatj/kernel-mod (17 stars, MIT, last commit 2020-02-15).

Abstract

Given two candidate models, and a set of target observations, we address the problem of measuring the relative goodness of fit of the two models. We propose two new statistical tests which are nonparametric, computationally efficient (runtime complexity is linear in the sample size), and interpretable. As a unique advantage, our tests can produce a set of examples (informative features) indicating the regions in the data domain where one model fits significantly better than the other. In a real-world problem of comparing GAN models, the test power of our new test matches that of the state-of-the-art test of relative goodness of fit, while being one order of magnitude faster.

View on arXivView PDF

Code Implementations(1)

wittawatj/kernel-modOfficial100%
MIT

NeurIPS 2018. Linear-time model comparison tests.

173Shell, PythonNov 3, 20186 years agoMIT
hypothesis-testinginterpretabilitykernel-methodsmachine-learningmodel-comparison

Similar Papers

A Linear-Time Kernel Goodness-of-Fit Test

82% match1 repo672017

Efficient Aggregated Kernel Tests using Incomplete U-statistics

81% match1 repo32022

A Bayesian Bradley-Terry model to compare multiple ML algorithms on multiple data sets

78% match1 repo52022

Robust Nonparametric Hypothesis Testing to Understand Variability in Training Neural Networks

77% match2023

Robustness Quantification and Uncertainty Quantification: Comparing Two Methods for Assessing the Reliability of Classifier Predictions

77% match2026

A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms

77% match2 repos62019

Cite this paper

@article{jitkrittum2018informative,
  title  = {Informative Features for Model Comparison},
  author = {Wittawat Jitkrittum and Heishiro Kanagawa and Patsorn Sangkloy and James Hays and Bernhard Schölkopf and Arthur Gretton},
  year   = {2018},
  eprint = {1810.11630},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1810.11630}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1810.11630.svg)](https://codeofpaper.com/paper/1810.11630)

Discussion