S^2-LBI: Stochastic Split Linearized Bregman Iterations for Parsimonious Deep Learning
share
This paper proposes a novel Stochastic Split Linearized Bregman Iteration (S^2-LBI) algorithm to efficiently train the deep network. The S^2-LBI introduces an iterative regularization path with structural sparsity. Our S^2-LBI combines the computational efficiency of the LBI, and model selection consistency in learning the structural sparsity. The computed solution path intrinsically enables us to enlarge or simplify a network, which theoretically, is benefited from the dynamics property of our S^2-LBI algorithm. The experimental results validate our S^2-LBI on MNIST and CIFAR-10 dataset. For example, in MNIST, we can either boost a network with only 1.5K parameters (1 convolutional layer of 5 filters, and 1 FC layer), achieves 98.40% recognition accuracy; or we simplify 82.5% of parameters in LeNet-5 network, and still achieves the 98.47% recognition accuracy. In addition, we also have the learning results on ImageNet, which will be added in the next version of our report.
READ FULL TEXT