research
∙
07/27/2023
The Marginal Value of Momentum for Small Learning Rate SGD
Momentum is known to accelerate the convergence of gradient descent in s...
share
research
∙
03/02/2023
Why (and When) does Local SGD Generalize Better than SGD?
Local SGD is a communication-efficient variant of SGD for large-scale tr...
research
∙
01/27/2023
Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing
It is believed that Gradient Descent (GD) induces an implicit bias towar...
research
∙
11/05/2022
New Definitions and Evaluations for Saliency Methods: Staying Intrinsic, Complete and Sound
Saliency methods compute heat maps that highlight portions of an input t...
research
∙
06/14/2022
Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction
Normalization layers (e.g., Batch Normalization, Layer Normalization) we...
research
∙
05/20/2022
On the SDEs and Scaling Rules for Adaptive Gradient Algorithms
Approximating Stochastic Gradient Descent (SGD) as a Stochastic Differen...
research
∙
10/26/2021
Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias
The generalization mystery of overparametrized deep nets has motivated e...
research
∙
12/17/2020
Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning
Matrix factorization is a simple and natural test-bed to investigate the...
research
∙
10/06/2020
Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate
Recent works (e.g., (Li and Arora, 2020)) suggest that the use of popula...
research
∙
06/13/2019
Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
Recent works on implicit regularization have shown that gradient descent...
research
∙
12/10/2018
Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
Batch Normalization (BN) has become a cornerstone of deep learning acros...
research
∙
08/31/2018
Single-Source Bottleneck Path Algorithm Faster than Sorting for Sparse Graphs
In a directed graph G=(V,E) with a capacity on every edge, a bottleneck ...
research
∙
05/07/2018