research
∙
03/30/2022
Transformer Language Models without Positional Encodings Still Learn Positional Information
Transformers typically require some form of positional encoding, such as...
share
research
∙
12/14/2021
Learning to Retrieve Passages without Supervision
Dense retrievers for open-domain question answering (ODQA) have been sho...
research
∙
12/29/2020
Transformer Feed-Forward Layers Are Key-Value Memories
Feed-forward layers constitute two-thirds of a transformer model's param...
research
∙
11/07/2019
Blockwise Self-Attention for Long Document Understanding
We present BlockBERT, a lightweight and efficient BERT model that is des...
research
∙
11/01/2019
Generalization through Memorization: Nearest Neighbor Language Models
We introduce kNN-LMs, which extend a pre-trained neural language model (...
research
∙
10/29/2019
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
We present BART, a denoising autoencoder for pretraining sequence-to-seq...
research
∙
07/26/2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Language model pretraining has led to significant performance gains but ...
research
∙
06/11/2019
What Does BERT Look At? An Analysis of BERT's Attention
Large pre-trained neural networks such as BERT have had great recent suc...
research
∙
05/25/2019
Are Sixteen Heads Really Better than One?
Attention is a powerful and ubiquitous mechanism for allowing neural mod...
research
∙
05/02/2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
In the last year, new models and methods for pretraining and transfer le...
research
∙
04/19/2019
Constant-Time Machine Translation with Conditional Masked Language Models
Most machine translation systems generate text autoregressively, by sequ...
research
∙
08/04/2018
code2seq: Generating Sequences from Structured Representations of Code
The ability to generate natural language sequences from source code snip...
research
∙
02/15/2014
