Order Optimal One-Shot Federated Learning for non-Convex Loss Functions
share
We consider the problem of federated learning in a one-shot setting in which there are m machines, each observing n samples function from an unknown distribution on non-convex loss functions. Let F:[-1,1]^d→ℝ be the expected loss function with respect to this unknown distribution. The goal is to find an estimate of the minimizer of F. Based on its observations, each machine generates a signal of bounded length B and sends it to a server. The sever collects signals of all machines and outputs an estimate of the minimizer of F. We propose a distributed learning algorithm, called Multi-Resolution Estimator for Non-Convex loss function (MRE-NC), whose expected error is bounded by max(1/√(n)(mB)^1/d, 1/√(mn)), up to polylogarithmic factors. We also provide a matching lower bound on the performance of any algorithm, showing that MRE-NC is order optimal in terms of n and m. Experiments on synthetic and real data show the effectiveness of MRE-NC in distributed learning of model's parameters for non-convex loss functions.
READ FULL TEXT