DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks

07/25/2019
by   Lin Zehui, et al.
0

Variants dropout methods have been designed for the fully-connected layer, convolutional layer and recurrent layer in neural networks, and shown to be effective to avoid overfitting. As an appealing alternative to recurrent and convolutional layers, the fully-connected self-attention layer surprisingly lacks a specific dropout method. This paper explores the possibility of regularizing the attention weights in Transformers to prevent different contextualized feature vectors from co-adaption. Experiments on a wide range of tasks show that DropAttention can improve performance and reduce overfitting.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset