Templating Shuffles

07/21/2022

∙

by   Qizhen Zhang, et al.

∙

0

∙

share

Cloud data centers are rapidly evolving. At the same time, large-scale data analytics applications require non-trivial performance tuning that is often specific to the applications, workloads, and data center infrastructure. We propose TeShu, which makes network shuffling an extensible unified service layer common to all data analytics. Since an optimal shuffle depends on a myriad of factors, TeShu introduces parameterized shuffle templates, instantiated by accurate and efficient sampling that enables TeShu to dynamically adapt to different application workloads and data center layouts. Our experimental results with real-world graph workloads show that TeShu efficiently enables shuffling optimizations that improve performance and adapt to a variety of scenarios.