Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting

05/20/2018
by   Hippolyt Ritter, et al.
0

We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the weights. The Laplace approximation requires calculating the Hessian around a mode, which is typically intractable for modern architectures. In order to make our method scalable, we leverage recent block-diagonal Kronecker factored approximations to the curvature. Our algorithm achieves over 90 dataset, substantially outperforming related methods for overcoming catastrophic forgetting.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset