Related papers: A Serverless Engine for High Energy Physics Distri…
In the era of data-driven science, conducting computational experiments that involve analysing large datasets using heterogeneous computational clusters, is part of the everyday routine for many scientists. Moreover, to ensure the…
Distributed ledgers are a new type of database technology that allows open access to data stored across distributed, decentralised, publicly maintained infrastructures. Current implementations of the such ledgers expect competition between…
The advent of big data and AI has precipitated a demand for computational frameworks that ensure real-time performance, accuracy, and privacy. While edge computing mitigates latency and privacy concerns, its scalability is constrained by…
The Large Hadron Collider (LHC), which collides protons at an energy of 14 TeV, produces hundreds of exabytes of data per year, making it one of the largest sources of data in the world today. At present it is not possible to even transfer…
These lectures were presented at the 2019 CERN Latin-American School of High Energy Physics. They were centered on the experimental methods used in hadron colliders to advance our understanding in the field of high energy particle physics.…
The CERN Large Hadron Collider (LHC) is designed to collide proton beams of unprecedented energy, in order to extend the frontiers of high-energy particle physics. During the first very successful running period in 2010--2013, the LHC was…
In recent years, there has been a surge in the adoption of serverless computing due to the ease of deployment, attractive pay-per-use pricing, and transparent horizontal auto-scaling. At the same time, infrastructure advancements such as…
Modern datacenters schedule heterogeneous workloads across geo-distributed sites with diverse compute capacities, electricity prices, and thermal conditions. Compute utilization, heat generation, cooling demand, and energy consumption are…
The planned high-luminosity upgrade of the Large Hadron Collider (LHC) at CERN will bring much higher data rates that are far above the capabilities of currently installed software-based data processing systems. Therefore, new methods must…
Data access is key to science driven by distributed high-throughput computing (DHTC), an essential technology for many major research projects such as High Energy Physics (HEP) experiments. However, achieving efficient data access becomes…
Distributed Machine Learning refers to the practice of training a model on multiple computers or devices that can be called nodes. Additionally, serverless computing is a new paradigm for cloud computing that uses functions as a…
Increasing popularity of the serverless computing approach has led to the emergence of new cloud infrastructures working in Container-as-a-Service (CaaS) model like AWS Fargate, Google Cloud Run, or Azure Container Instances. They introduce…
We review the status of, and prospects for, real-time data processing for collider experiments in experimental High Energy Physics. We discuss the historical evolution of data rates and volumes in the field and place them in the context of…
In this chapter we will argue that studying such multi-scale multi-science systems gives rise to inherently hybrid models containing many different algorithms best serviced by different types of computing environments (ranging from…
A new class of Second generation high-performance computing applications with heterogeneous, dynamic and data-intensive properties have an extended set of requirements, which cover application deployment, resource allocation, -control, and…
The challenge to fully exploit the potential of existing and upcoming scientific instruments like large single-dish radio telescopes is to process the collected massive data effectively and efficiently. As a "quasi 2D stencil computation"…
In this work we demonstrate that significant gains in performance and data efficiency can be achieved in High Energy Physics (HEP) by moving beyond the standard paradigm of sequential optimization or reconstruction and analysis components.…
The Production and Distributed Analysis (PanDA) system, originally developed for the ATLAS experiment at the CERN Large Hadron Collider (LHC), has evolved into a robust platform for orchestrating large-scale workflows across distributed…
Modern database clusters entail two levels of networks: connecting CPUs and NUMA regions inside a single server in the small and multiple servers in the large. The huge performance gap between these two types of networks used to slow down…
Serverless computing has matured into an effective execution model for edge cloud environments, enabling function level decomposition, demand driven scaling, and workflow execution across stable, well provisioned infrastructure. This…