research
∙
03/29/2022
Training Compute-Optimal Large Language Models
We investigate the optimal model size and number of tokens for training ...
share
research
∙
12/08/2021
Improving language models by retrieving from trillions of tokens
We enhance auto-regressive language models by conditioning on document c...
research
∙
06/07/2021
Top-KAST: Top-K Always Sparse Training
Sparse neural networks are becoming increasingly important as the field ...
research
∙
07/07/2020
Do Transformers Need Deep Long-Range Memory
Deep attention models have advanced the modelling of sequential data acr...
research
∙
11/13/2019
Compressive Transformers for Long-Range Sequence Modelling
We present the Compressive Transformer, an attentive sequence model whic...
research
∙
10/13/2019
Stabilizing Transformers for Reinforcement Learning
Owing to their ability to both effectively integrate information over lo...
research
∙
10/07/2019
Meta-Learning Deep Energy-Based Memory Models
We study the problem of learning associative memory – a system which is ...
research
∙
09/26/2019
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
Some of the most successful applications of deep reinforcement learning ...
research
∙
06/10/2019
Meta-Learning Neural Bloom Filters
There has been a recent trend in training neural networks to replace dat...
research
∙
03/27/2018
Fast Parametric Learning with Activation Memorization
Neural networks trained with backpropagation often struggle to identify ...
research
∙
02/28/2018
Memory-based Parameter Adaptation
Deep neural networks have excelled on a wide range of problems, from vis...
research
∙
10/27/2016