research
∙
06/27/2023
Length Generalization in Arithmetic Transformers
We examine how transformers cope with two challenges: learning basic int...
share
research
∙
05/13/2023
Depth Dependence of μP Learning Rates in ReLU MLPs
In this short note we consider random fully connected ReLU networks of w...
research
∙
10/13/2022
Vision Transformers provably learn spatial structure
Vision Transformers (ViTs) have achieved comparable or superior performa...
research
∙
10/09/2022
Dissecting adaptive methods in GANs
Adaptive methods are a crucial component widely used for training genera...
research
∙
07/13/2022
Towards understanding how momentum improves generalization in deep learning
Stochastic gradient descent (SGD) with momentum is widely used for train...
research
∙
02/02/2021
Depth separation beyond radial functions
High-dimensional depth separation results for neural networks show that ...
research
∙
01/26/2021
Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization
We introduce MADGRAD, a novel optimization method in the family of AdaGr...
research
∙
10/20/2020
Dual Averaging is Surprisingly Effective for Deep Learning Optimization
First-order stochastic optimization methods are currently the most widel...
research
∙
06/10/2020
Auction learning as a two-player game
Designing an incentive compatible auction that maximizes expected revenu...
research
∙
03/02/2020
A Permutation-Equivariant Neural Network Architecture For Auction Design
Designing an incentive compatible auction that maximizes expected revenu...
research
∙
02/14/2020
A mean-field analysis of two-player zero-sum games
Finding Nash equilibria in two-player zero-sum continuous games is a cen...
research
∙
07/31/2019
Towards closing the gap between the theory and practice of SVRG
Among the very first variance reduced stochastic methods for solving the...
research
∙
05/29/2019
Extra-gradient with player sampling for provable fast convergence in n-player games
Data-driven model training is increasingly relying on finding Nash equil...
research
∙
02/05/2019
Global convergence of neuron birth-death dynamics
Neural networks with a large number of parameters admit a mean-field des...
research
∙
06/11/2018
