Workflows Community Summit 2024: Future Trends and Challenges in Scientific Workflows
Abstract
The Workflows Community Summit gathered 111 participants from 18 countries to discuss emerging trends and challenges in scientific workflows, focusing on six key areas: time-sensitive workflows, AI-HPC convergence, multi-facility workflows, heterogeneous HPC environments, user experience, and FAIR computational workflows. The integration of AI and exascale computing has revolutionized scientific workflows, enabling higher-fidelity models and complex, time-sensitive processes, while introducing challenges in managing heterogeneous environments and multi-facility data dependencies. The rise of large language models is driving computational demands to zettaflop scales, necessitating modular, adaptable systems and cloud-service models to optimize resource utilization and ensure reproducibility. Multi-facility workflows present challenges in data movement, curation, and overcoming institutional silos, while diverse hardware architectures require integrating workflow considerations into early system design and developing standardized resource management tools. The summit emphasized improving user experience in workflow systems and ensuring FAIR workflows to enhance collaboration and accelerate scientific discovery. Key recommendations include developing standardized metrics for time-sensitive workflows, creating frameworks for cloud-HPC integration, implementing distributed-by-design workflow modeling, establishing multi-facility authentication protocols, and accelerating AI integration in HPC workflow management. The summit also called for comprehensive workflow benchmarks, workflow-specific UX principles, and a FAIR workflow maturity model, highlighting the need for continued collaboration in addressing the complex challenges posed by the convergence of AI, HPC, and multi-facility research environments.
Cite
@article{arxiv.2410.14943,
title = {Workflows Community Summit 2024: Future Trends and Challenges in Scientific Workflows},
author = {Rafael Ferreira da Silva and Deborah Bard and Kyle Chard and Shaun de Witt and Ian T. Foster and Tom Gibbs and Carole Goble and William Godoy and Johan Gustafsson and Utz-Uwe Haus and Stephen Hudson and Shantenu Jha and Laila Los and Drew Paine and Frédéric Suter and Logan Ward and Sean Wilkinson and Marcos Amaris and Yadu Babuji and Jonathan Bader and Riccardo Balin and Daniel Balouek and Sarah Beecroft and Khalid Belhajjame and Rajat Bhattarai and Wes Brewer and Paul Brunk and Silvina Caino-Lores and Henri Casanova and Daniela Cassol and Jared Coleman and Taina Coleman and Iacopo Colonnelli and Anderson Andrei Da Silva and Daniel de Oliveira and Pascal Elahi and Nour Elfaramawy and Wael Elwasif and Brian Etz and Thomas Fahringer and Wesley Ferreira and Rosa Filgueira and Jacob Fosso Tande and Luiz Gadelha and Andy Gallo and Daniel Garijo and Yiannis Georgiou and Philipp Gritsch and Patricia Grubel and Amal Gueroudji and Quentin Guilloteau and Carlo Hamalainen and Rolando Hong Enriquez and Lauren Huet and Kevin Hunter Kesling and Paula Iborra and Shiva Jahangiri and Jan Janssen and Joe Jordan and Sehrish Kanwal and Liliane Kunstmann and Fabian Lehmann and Ulf Leser and Chen Li and Peini Liu and Jakob Luettgau and Richard Lupat and Jose M. Fernandez and Ketan Maheshwari and Tanu Malik and Jack Marquez and Motohiko Matsuda and Doriana Medic and Somayeh Mohammadi and Alberto Mulone and John-Luke Navarro and Kin Wai Ng and Klaus Noelp and Bruno P. Kinoshita and Ryan Prout and Michael R. Crusoe and Sashko Ristov and Stefan Robila and Daniel Rosendo and Billy Rowell and Jedrzej Rybicki and Hector Sanchez and Nishant Saurabh and Sumit Kumar Saurav and Tom Scogland and Dinindu Senanayake and Woong Shin and Raul Sirvent and Tyler Skluzacek and Barry Sly-Delgado and Stian Soiland-Reyes and Abel Souza and Renan Souza and Domenico Talia and Nathan Tallent and Lauritz Thamsen and Mikhail Titov and Benjamin Tovar and Karan Vahi and Eric Vardar-Irrgang and Edite Vartina and Yuandou Wang and Merridee Wouters and Qi Yu and Ziad Al Bkhetan and Mahnoor Zulfiqar},
journal= {arXiv preprint arXiv:2410.14943},
year = {2024}
}