Middleware Building Blocks for Workflow Systems

03/24/2019
by   Matteo Turilli, et al.
0

This paper describes a building blocks approach to the design of scientific workflow systems. We discuss RADICAL-Cybertools as one implementation of the building blocks concept, showing how they are designed and developed in accordance with this approach. Four case studies are presented, discussing how RADICAL-Cybertools are integrated with existing workflow, workload, and general purpose computing systems to support the execution of scientific workflows. This paper offers three main contributions: (i) showing the relevance of the design principles of self-sufficiency, interoperability, composability and extensibility for middleware to support scientific workflows on high performance computing machines; (ii) illustrating a set of building blocks that enable multiple points of integration, which results in design flexibility and functional extensibility, as well as providing a level of "unification" in the conceptual reasoning across otherwise very different tools and systems; and (iii) showing how these building blocks have been used to develop and integrate workflow systems.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset