Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study)

Michael Weiss, Paolo Tonella
5/2/2022
cs.LGcs.AIcs.SE

Official code: testingautomated-usi/dnn-tip (19 stars, MIT, last commit 2023-02-03). 1 more implementation below.

Abstract

Test Input Prioritizers (TIP) for Deep Neural Networks (DNN) are an important technique to handle the typically very large test datasets efficiently, saving computation and labeling costs. This is particularly true for large-scale, deployed systems, where inputs observed in production are recorded to serve as potential test or training data for the next versions of the system. Feng et. al. propose DeepGini, a very fast and simple TIP, and show that it outperforms more elaborate techniques such as neuron- and surprise coverage. In a large-scale study (4 case studies, 8 test datasets, 32'200 trained models) we verify their findings. However, we also find that other comparable or even simpler baselines from the field of uncertainty quantification, such as the predicted softmax likelihood or the entropy of the predicted softmax likelihoods perform equally well as DeepGini.

View on arXivView PDF

Code Implementations(2)

testingautomated-usi/dnn-tipOfficial100%
MIT

A collection of dnn test input prioritizers often used as benchmarks in recent literature.

192PythonApr 5, 20223 years agoMIT
testingautomated-usi/simple-tipOfficial100%
MIT

This is the reproduction package of the paper Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning by M.Weiss and P.Tonella, published at ISSTA 2022

51Python, DockerfileApr 28, 20224 years agoMIT
active-learningdeep-learningissta-2022machine-learningreplication-study+2 more

Similar Papers

LiBaGS: Lightweight Boundary Gap Synthesis for Targeted Synthetic Data Selection

80% match2026

SAVA: Scalable Learning-Agnostic Data Valuation

80% match2024

Rethinking Epistemic and Aleatoric Uncertainty for Active Open-Set Annotation: An Energy-Based Approach

78% match1 repo92025

Human Uncertainty-Aware Data Selection and Automatic Labeling in Visual Question Answering

78% match2025

General Information Metrics for Improving AI Model Training Efficiency

78% match2025

Trust the Unreliability: Inward Backward Dynamic Unreliability Driven Coreset Selection for Medical Image Classification

78% match2026

Cite this paper

@article{weiss2022simple,
  title  = {Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study)},
  author = {Michael Weiss and Paolo Tonella},
  year   = {2022},
  eprint = {2205.00664},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2205.00664}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2205.00664.svg)](https://codeofpaper.com/paper/2205.00664)

Discussion