research
∙
08/11/2023
Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reduction
The recently proposed stochastic Polyak stepsize (SPS) and stochastic li...
share
research
∙
07/12/2023
Locally Adaptive Federated Learning via Stochastic Polyak Stepsizes
State-of-the-art federated learning algorithms such as FedAvg require ca...
research
∙
06/23/2023
Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity
In federated learning, data heterogeneity is a critical challenge. A str...
research
∙
06/08/2023
Communication-Efficient Gradient Descent-Accent Methods for Distributed Variational Inequalities: Unified Analysis and Local Updates
Distributed and federated learning algorithms and techniques associated ...
research
∙
05/30/2023
Shuffle SGD is Always Better than SGD: Improved Analysis of SGD with Arbitrary Data Orders
Stochastic Gradient Descent (SGD) algorithms are widely used in optimizi...
research
∙
05/02/2023
Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees
Gradient clipping is a popular modification to standard (stochastic) gra...
research
∙
01/03/2023
Decentralized Gradient Tracking with Local Steps
Gradient tracking (GT) is an algorithm designed for solving decentralize...
research
∙
12/05/2022
Partial Variance Reduction improves Non-Convex Federated learning on heterogeneous data
Data heterogeneity across clients is a key challenge in federated learni...
research
∙
06/16/2022
Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning
We study the asynchronous stochastic gradient descent algorithm for dist...
research
∙
04/13/2022
Data-heterogeneity-aware Mixing for Decentralized Learning
Decentralized learning provides an effective framework to train machine ...
research
∙
02/18/2022
Tackling benign nonconvexity with smoothing and stochastic gradients
Non-convex optimization problems are ubiquitous in machine learning, esp...
research
∙
02/08/2022
An Improved Analysis of Gradient Tracking for Decentralized Machine Learning
We consider decentralized machine learning over a network where the trai...
research
∙
12/09/2021
The Peril of Popular Deep Learning Uncertainty Estimation Methods
Uncertainty estimation (UE) techniques – such as the Gaussian process (G...
research
∙
11/10/2021
Linear Speedup in Personalized Collaborative Learning
Personalization in federated learning can improve the accuracy of a mode...
research
∙
10/11/2021
ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training
Federated learning is a powerful distributed learning scheme that allows...
research
∙
10/08/2021
RelaySum for Decentralized Deep Learning on Heterogeneous Data
In decentralized machine learning, workers compute model updates on thei...
research
∙
09/06/2021
On Second-order Optimization Methods for Federated Learning
We consider federated learning (FL), where the training data is distribu...
research
∙
08/18/2021
Semantic Perturbations with Normalizing Flows for Improved Generalization
Data augmentation is a widely adopted technique for avoiding overfitting...
research
∙
06/16/2021
Simultaneous Training of Partially Masked Neural Networks
For deploying deep learning models to lower end devices, it is necessary...
research
∙
06/15/2021
Decentralized Local Stochastic Extra-Gradient for Variational Inequalities
We consider decentralized stochastic variational inequalities where the ...
research
∙
03/03/2021
Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates
It has been experimentally observed that the efficiency of distributed t...
research
∙
02/09/2021
Consensus Control for Decentralized Deep Learning
Decentralized training of deep learning models enables on-device learnin...
research
∙
02/09/2021
Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
Decentralized training of deep learning models is a key element for enab...
research
∙
11/03/2020
A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!
Decentralized optimization methods enable on-device training of machine ...
research
∙
09/04/2020
On Communication Compression for Distributed Optimization on Heterogeneous Data
Lossy gradient compression, with either unbiased or biased compressors, ...
research
∙
08/08/2020
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
Federated learning is a challenging optimization problem due to the hete...
research
∙
07/31/2020
Analysis of SGD with Biased Gradient Estimators
We analyze the complexity of biased stochastic gradient methods (SGD), w...
research
∙
06/12/2020
Dynamic Model Pruning with Feedback
Deep neural networks often have millions of parameters. This can hinder ...
research
∙
06/12/2020
Ensemble Distillation for Robust Model Fusion in Federated Learning
Federated Learning (FL) is a machine learning setting where many devices...
research
∙
06/10/2020
Extrapolation for Large-batch Training in Deep Learning
Deep learning networks are typically trained by Stochastic Gradient Desc...
research
∙
03/23/2020
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
Decentralized stochastic optimization methods have gained a lot of atten...
research
∙
02/18/2020
Is Local SGD Better than Minibatch SGD?
We study local SGD (also known as parallel SGD and federated averaging),...
research
∙
12/10/2019
Advances and Open Problems in Federated Learning
Federated learning (FL) is a machine learning setting where many clients...
research
∙
10/14/2019
SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning
Federated learning is a key scenario in modern large-scale machine learn...
research
∙
09/11/2019
The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
We analyze (stochastic) gradient descent (SGD) with delayed updates on s...
research
∙
07/22/2019
Decentralized Deep Learning with Arbitrary Communication Compression
Decentralized training of deep learning models is a key element for enab...
research
∙
07/09/2019
Unified Optimal Analysis of the (Stochastic) Gradient Method
In this note we give a simple proof for the convergence of stochastic gr...
research
∙
02/01/2019
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
We consider decentralized stochastic optimization with the objective fun...
research
∙
01/28/2019
Error Feedback Fixes SignSGD and other Gradient Compression Schemes
Sign-based algorithms (e.g. signSGD) have been proposed as a biased grad...
research
∙
10/16/2018
Efficient Greedy Coordinate Descent for Composite Problems
Coordinate descent with random coordinate selection is the current state...
research
∙
09/20/2018
Sparsified SGD with Memory
Huge scale machine learning problems are nowadays tackled by distributed...
research
∙
08/22/2018
Don't Use Large Mini-Batches, Use Local SGD
Mini-batch stochastic gradient methods are the current state of the art ...
research
∙
06/01/2018
Global linear convergence of Newton's method without strong-convexity or Lipschitz gradients
We show that Newton's method converges globally at a linear rate for obj...
research
∙
05/24/2018
Local SGD Converges Fast and Communicates Little
Mini-batch stochastic gradient descent (SGD) is the state of the art in ...
research
∙
05/02/2018
SVRG meets SAGA: k-SVRG --- A Tale of Limited Memory
In recent years, many variance reduced algorithms for empirical risk min...
research
∙
03/26/2018