What can we learn from gradients?
share
Recent work (<cit.>) has shown that it is possible to reconstruct the input (image) from the gradient of a neural network. In this paper, our aim is to better understand the limits to reconstruction and to speed up image reconstruction by imposing prior image information and improved initialization. Firstly, we show that for the non-linear neural network, gradient-based reconstruction approximates to solving a high-dimension linear equations for both fully-connected neural network and convolutional neural network. Exploring the theoretical limits of input reconstruction, we show that a fully-connected neural network with a one hidden node is enough to reconstruct a single input image, regardless of the number of nodes in the output layer. Then we generalize this result to a gradient averaged over mini-batches of size B. In this case, the full mini-batch can be reconstructed in a fully-connected network if the number of hidden units exceeds B. For a convolutional neural network, the required number of filters in the first convolutional layer again is decided by the batch size B, however, in this case, input width d and the width after filter d^' also play the role h=(d/d^')^2BC, where C is channel number of input. Finally, we validate and underpin our theoretical analysis on bio-medical data (fMRI, ECG signals, and cell images) and on benchmark data (MNIST, CIFAR100, and face images).
READ FULL TEXT