research
∙
06/01/2023
Transformers learn to implement preconditioned gradient descent for in-context learning
Motivated by the striking ability of transformers for in-context learnin...
share
research
∙
05/28/2023
On the impact of activation and normalization in obtaining isometric embeddings at initialization
In this paper, we explore the structure of the penultimate Gram matrix i...
research
∙
02/09/2023
Efficient displacement convex optimization with particle gradient descent
Particle gradient descent, which uses particles to represent a probabili...
research
∙
05/25/2022
Entropy Maximization with Depth: A Variational Principle for Random Neural Networks
To understand the essential role of depth in neural networks, we investi...
research
∙
04/16/2022
Polynomial-time sparse measure recovery
How to recover a probability measure with sparse support from particular...
research
∙
06/07/2021
Batch Normalization Orthogonalizes Representations in Deep Random Networks
This paper underlines a subtle property of batch-normalization (BN): Suc...
research
∙
02/23/2021
Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization
Viewing optimization methods as numerical integrators for ordinary diffe...
research
∙
10/31/2019
Mixing of Stochastic Accelerated Gradient Descent
We study the mixing properties for stochastic accelerated gradient desce...
research
∙
05/27/2018
Towards a Theoretical Understanding of Batch Normalization
Normalization techniques such as Batch Normalization have been applied v...
research
∙
05/15/2018
Local Saddle Point Optimization: A Curvature Exploitation Approach
Gradient-based optimization methods are the most popular choice for find...
research
∙
03/15/2018
Escaping Saddles with Stochastic Gradients
We analyze the variance of stochastic gradients along negative curvature...
research
∙
05/12/2014