research
∙
09/11/2023
Uncovering mesa-optimization algorithms in Transformers
Transformers have become the dominant model in deep learning, but the re...
share
research
∙
09/04/2023
Gated recurrent neural networks discover attention
Recent architectural developments have enabled recurrent neural networks...
research
∙
05/25/2023
Online learning of long-range dependencies
Online learning holds the promise of enabling efficient long-term credit...
research
∙
09/15/2022
Random initialisations performing above chance and how to find them
Neural networks trained with stochastic gradient descent (SGD) starting ...
research
∙
07/04/2022
The least-control principle for learning at equilibrium
Equilibrium systems are a powerful way to express neural computations. A...
research
∙
05/06/2022
Beyond backpropagation: implicit gradients for bilevel optimization
This paper reviews gradient-based techniques to solve bilevel optimizati...
research
∙
10/27/2021
Learning where to learn: Gradient sparsity in meta and continual learning
Finding neural network weights that generalize well from small datasets ...
research
∙
04/04/2021