Measuring Uncertainty Disentanglement Error in Classification

Abstract

Current methods for disentangling aleatoric and epistemic uncertainty in classification with Bayesian Neural Networks have been receiving strong criticism. Information Theoretic measures for aleatoric and epistemic uncertainty are not independent, due to the additivity assumption. However, these investigations do not consider Gaussian Logits, an alternative approach that gets less attention. In this paper, we present a set of three experiments that manipulate the aleatoric and epistemic uncertainty in isolation to benchmark the quality of disentanglement using multiple datasets. Based on these experiments we define the Disentanglement Error as a metric for the quality of disentanglement. We evaluate Information Theoretic and Gaussian Logits disentangling over multiple Bayesian Neural Networks approximations and show that Deep Ensembles with Information Theoretic disentanglement have the best Disentanglement Error, but there is still room for improvement.