NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks

Fahim Dalvi, Avery Nortonsmith, D. Anthony Bau, Yonatan Belinkov, Hassan Sajjad +2 more
12/21/2018
cs.CL

Official code: fdalvi/NeuroX (109 stars, BSD-3-Clause, last commit 2023-10-04).

Abstract

We present a toolkit to facilitate the interpretation and understanding of neural network models. The toolkit provides several methods to identify salient neurons with respect to the model itself or an external task. A user can visualize selected neurons, ablate them to measure their effect on the model accuracy, and manipulate them to control the behavior of the model at the test time. Such an analysis has a potential to serve as a springboard in various research directions, such as understanding the model, better architectural choices, model distillation and controlling data biases.

View on arXivView PDF

Code Implementations(1)

fdalvi/NeuroXOfficial100%
BSD-3-Clause

A Python library that encapsulates various methods for neuron interpretation and analysis in Deep NLP models.

10927Shell, PythonAug 16, 20183 years agoBSD-3-Clause
explainable-ainatural-language-processingneuronsnlpnlp-machine-learning

Similar Papers

NeuroX Library for Neuron Analysis of Deep NLP Models

89% match1 repo1092023

NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactions

85% match1 repo72025

Suitability of KANs for Computer Vision: A preliminary investigation

84% match2024

Using ScrutinAI for Visual Inspection of DNN Performance in a Medical Use Case

84% match2023

NxPlain: Web-based Tool for Discovery of Latent Concepts

84% match2023

SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding

83% match2026

Cite this paper

@article{dalvi2018neurox,
  title  = {NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks},
  author = {Fahim Dalvi and Avery Nortonsmith and D. Anthony Bau and Yonatan Belinkov and Hassan Sajjad and Nadir Durrani and James Glass},
  year   = {2018},
  eprint = {1812.09359},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1812.09359}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1812.09359.svg)](https://codeofpaper.com/paper/1812.09359)

Discussion