Sampling Free Epistemic Uncertainty Through Approximated Variance Propagation

Noise injection methods such as dropout are popular ways of implicitly capturing epistemic uncertainty in neural networks. Usually, noise injection is applied during training and inference. In the training phase, the networks learns to reduce the variance that is introduced by noise injection within the data distribution. Running inference several times on the same input with noise injection enabled makes it possible to estimate the remaining uncertainty, which will be mostly epistemic. Applying noise injection during inference, however, increases the computational costs and can be too slow for real-time applications.

In this talk we will discuss how to use approximate variance propagation as a faster and deterministic way of capturing the variance that is introduced by noise injection during inference.

References

[Pos19S]

Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation, Janis Postels, Francesco Ferroni, Huseyin Coskun, Nassir Navab, Federico Tombari.

Dec 2019

We present a sampling-free approach for computing the epistemic uncertainty of a neural network. Epistemic uncertainty is an important quantity for the deployment of deep neural networks in safety-critical applications, since it represents how much one can trust predictions on new data. Recently promising works were proposed using noise injection combined with Monte-Carlo sampling at inference …

[Gal16D]

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, Yarin Gal, Zoubin Ghahramani.

Oct 2016

Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with a prohibitive computational cost. In this paper we develop a new theoretical framework casting …

[Bra20S]

Single Shot MC Dropout Approximation, Kai Brach, Beate Sick, Oliver Dürr.

Jul 2020

Deep neural networks (DNNs) are known for their high prediction performance, especially in perceptual tasks such as object recognition or autonomous driving. Still, DNNs are prone to yield unreliable predictions when encountering completely new situations without indicating their uncertainty. Bayesian variants of DNNs (BDNNs), such as MC dropout BDNNs, do provide uncertainty measures. However, …

[Wan13F]

Fast dropout training, Sida Wang, Christopher Manning.

May 2013

Preventing feature co-adaptation by encouraging independent contributions from different features often improves classification and regression performance. Dropout training (Hinton et al., 2012) does this by randomly dropping out (zeroing) hidden units and input features during training of neural networks. However, repeatedly sampling a random subset of input features makes training much slower. …

References

In this series →