Abstract

With the increased prevalence of employing workflows for scientific computing and a push towards exascale computing, it has become paramount that we are able to analyze characteristics of scientific applications to better understand their impact on the underlying infrastructure and vice-versa. Such analysis can help drive the design, development, and optimization of these next generation systems and solutions. In this paper, we present the architecture, integrated with existing well-established and newly developed tools, to collect online performance statistics of workflow executions from various, heterogeneous sources and publish them in a distributed database (Elasticsearch). Using this architecture, we are able to correlate online workflow performance data, with data from the underlying infrastructure, and present them in a useful and intuitive way via an online dashboard. We have validated our approach by …

Metadata

publication

Future Generation Computer Systems 117, 387-400, 2021

year

2021

publication date

2021/4/1

authors

George Papadimitriou, Cong Wang, Karan Vahi, Rafael Ferreira da Silva, Anirban Mandal, Zhengchun Liu, Rajiv Mayani, Mats Rynge, Mariam Kiran, Vickie E Lynch, Rajkumar Kettimuthu, Ewa Deelman, Jeffrey S Vetter, Ian Foster

link

https://www.sciencedirect.com/science/article/pii/S0167739X20330570

resource_link

https://www.sciencedirect.com/science/article/am/pii/S0167739X20330570

journal

Future Generation Computer Systems

volume

117

pages

387-400

publisher

North-Holland