Abstract

Workflow management systems (WMS) provide a robust solution for automating and ensuring the reproducibility of scientific and engineering experiments. However, reproducing machine learning (ML) experiments requires replicating every aspect of the process, including code implementation, workflow execution, data, and the execution environment. Traditionally, tracking these components is done manually, if at all, before execution. In this work, we propose an approach for on-demand and dynamic tracking of ML workflows. Our approach extends the ML workflow automatically and introduces steps for tracking, organizing, and versioning all elements, such as code, data, the main workflow steps and the execution environment for each job. This tracking approach includes two modes: a custom mode, where user-tagged elements will be tracked by the WMS, and an automatic mode, where the WMS automatically …

Metadata

publication

2024 IEEE 20th International Conference on e-Science (e-Science), 1-10, 2024

year

2024

publication date

2024/9/16

authors

Hamza Safri, George Papadimitriou, Ewa Deelman

link

https://ieeexplore.ieee.org/abstract/document/10678658/

conference

2024 IEEE 20th International Conference on e-Science (e-Science)

pages

1-10

publisher

IEEE