Measuring Information Transfer in Neural Networks

09/16/2020
by   Xiao Zhang, et al.
10

Estimation of the information content in a neural network model can be prohibitive, because of difficulty in finding an optimal codelength of the model. We propose to use a surrogate measure to bypass directly estimating model information. The proposed Information Transfer (L_IT) is a measure of model information based on prequential coding. L_IT is theoretically connected to model information, and is consistently correlated with model information in experiments. We show that L_IT can be used as a measure of generalizable knowledge in a model or a dataset. Therefore, L_IT can serve as an analytical tool in deep learning. We apply L_IT to compare and dissect information in datasets, evaluate representation models in transfer learning, and analyze catastrophic forgetting and continual learning algorithms. L_IT provides an informational perspective which helps us discover new insights into neural network learning.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset