Zero-shifting Technique for Deep Neural Network Training on Resistive Cross-point Arrays
share
A resistive memory device-based computing architecture is one of the promising platforms for energy-efficient Deep Neural Network (DNN) accelerators. The key technical challenge in realizing neural network training accelerator with resistive memory devices is to accumulate the gradient information in an unbiased way. Therefore, the device switching characteristics play a critical role in determining the training performance of DNNs. Unlike the digital numbers in software which can be assigned and accessed with desired accuracy, numbers stored in resistive memory devices in the form of resistance can only be manipulated following by the physics of the device. Therefore, additional techniques and algorithm-level remedies are required to achieve the best possible performance in resistive memory device-based DNN accelerators. In this paper, we analyze asymmetric conductance modulation characteristics in RRAM by Soft-bound synapse model and present an in-depth analysis on the relationship between device characteristics and DNN model accuracy using a 3-layer DNN trained on the MNIST dataset. We show that the imbalance between up and down update leads to a poor network performance. We introduce a concept of symmetry point and propose a zero-shifting technique which is a method to compensate imbalance by programming the reference device and changing the zero value point of the weight. By using this zero-shifting method, we show that network performance dramatically improves for imbalanced synapse devices.
READ FULL TEXT