research
∙
09/11/2023
Uncovering mesa-optimization algorithms in Transformers
Transformers have become the dominant model in deep learning, but the re...
share
research
∙
01/11/2023
Continual Few-Shot Learning Using HyperTransformers
We focus on the problem of learning without forgetting from multiple tas...
research
∙
01/05/2023
Training trajectories, mini-batch losses and the curious role of the learning rate
Stochastic gradient descent plays a fundamental role in nearly all appli...
research
∙
12/15/2022
Transformers learn in-context by gradient descent
Transformers have become the state-of-the-art neural network architectur...
research
∙
11/28/2022
Decentralized Learning with Multi-Headed Distillation
Decentralized learning with private data is a central problem in machine...
research
∙
03/29/2022
Fine-tuning Image Transformers using Learnable Memory
In this paper we propose augmenting Vision Transformer models with learn...
research
∙
01/13/2022
GradMax: Growing Neural Networks using Gradient Information
The architecture and the parameters of neural networks are often optimiz...
research
∙
01/11/2022
HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning
In this work we propose a HyperTransformer, a transformer-based model fo...
research
∙
04/10/2021
Meta-Learning Bidirectional Update Rules
In this paper, we introduce a new type of generalized neural network whe...
research
∙
11/06/2020
Underspecification Presents Challenges for Credibility in Modern Machine Learning
ML models often exhibit unexpectedly poor behavior when they are deploye...
research
∙
06/26/2019
