Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
share
We introduce a variational Bayesian neural network where the parameters are governed via a probability distribution on random matrices. Specifically, we employ a matrix variate Gaussian gupta1999matrix parameter posterior distribution where we explicitly model the covariance among the input and output dimensions of each layer. Furthermore, with approximate covariance matrices we can achieve a more efficient way to represent those correlations that is also cheaper than fully factorized parameter posteriors. We further show that with the "local reprarametrization trick" kingma2015variational on this posterior distribution we arrive at a Gaussian Process rasmussen2006gaussian interpretation of the hidden units in each layer and we, similarly with gal2015dropout, provide connections with deep Gaussian processes. We continue in taking advantage of this duality and incorporate "pseudo-data" snelson2005sparse in our model, which in turn allows for more efficient sampling while maintaining the properties of the original model. The validity of the proposed approach is verified through extensive experiments.
READ FULL TEXT