Try Depth Instead of Weight Correlations: Mean-field is a Less Restrictive Assumption for Deeper Networks

02/10/2020
by   Sebastian Farquhar, et al.
13

We challenge the longstanding assumption that the mean-field approximation for variational inference in Bayesian neural networks is severely restrictive. We argue mathematically that full-covariance approximations only improve the ELBO if they improve the expected log-likelihood. We further show that deeper mean-field networks are able to express predictive distributions approximately equivalent to shallower full-covariance networks. We validate these observations empirically, demonstrating that deeper models decrease the divergence between diagonal- and full-covariance Gaussian fits to the true posterior.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset