The Holdout Randomization Test for Feature Selection in Black Box Models

Wesley Tansey, Victor Veitch, Haoran Zhang, Raul Rabadan, David M. Blei
11/1/2018
stat.ME

Official code: tansey/hrt (22 stars, MIT, last commit 2021-03-19).

Abstract

We propose the holdout randomization test (HRT), an approach to feature selection using black box predictive models. The HRT is a specialized version of the conditional randomization test (CRT; Candes et al., 2018) that uses data splitting for feasible computation. The HRT works with any predictive model and produces a valid $p$-value for each feature. To make the HRT more practical, we propose a set of extensions to maximize power and speed up computation. In simulations, these extensions lead to greater power than a competing knockoffs-based approach, without sacrificing control of the error rate. We apply the HRT to two case studies from the scientific literature where heuristics were originally used to select important features for predictive models. The results illustrate how such heuristics can be misleading relative to principled methods like the HRT. Code is available at https://github.com/tansey/hrt.

View on arXivView PDF

Code Implementations(1)

tansey/hrtOfficial100%
MIT

The holdout randomization test: feature selection using black box predictive models

227R, PythonNov 1, 20185 years agoMIT

Similar Papers

When Features Beat Noise: A Feature Selection Technique Through Noise-Based Hypothesis Testing

82% match2025

Learning to Increase the Power of Conditional Randomization Tests

82% match2022

Statistical Test for Auto Feature Engineering by Selective Inference

79% match1 repo12024

A Survey on Active Feature Acquisition Strategies

79% match2025

Statistical Test for Feature Selection Pipelines by Selective Inference

78% match1 repo32024

Stabilizing black-box algorithms through task-oriented randomization

78% match2026

Cite this paper

@article{tansey2018holdout,
  title  = {The Holdout Randomization Test for Feature Selection in Black Box Models},
  author = {Wesley Tansey and Victor Veitch and Haoran Zhang and Raul Rabadan and David M. Blei},
  year   = {2018},
  eprint = {1811.00645},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1811.00645}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1811.00645.svg)](https://codeofpaper.com/paper/1811.00645)

Discussion