Communication-free Massively Distributed Graph Generation
share
Analyzing massive complex networks yields promising insights about our everyday lives. Building scalable algorithms to do that is a challenging task that requires a careful analysis and extensive evaluation. However, engineering such algorithms is often hindered by the scarcity of publicly available datasets. Network generators serve as a tool to alleviate this problem by providing synthetic instances with controllable parameters. However, many network generators fail to provide instances on a massive scale due to their sequential nature or resource constraints. Additionally, truly scalable network generators are few and often limited in their realism. In this work, we present novel generators for a variety of network models commonly found in practice. By making use of pseudorandomization and divide-and-conquer schemes, our generators follow a communication-free paradigm, i.e. they require no communication. The resulting generators are often embarrassingly parallel and have a near optimal scaling behavior. Overall, we are able to generate instances of up to 2^43 vertices and 2^47 edges in less than 22 minutes on 32768 processors. Therefore, our generators allow new graph families to be used on an unprecedented scale.
READ FULL TEXT