Related papers: Coffea-casa: an analysis facility prototype
The large data volumes expected from the High Luminosity LHC (HL-LHC) present challenges to existing paradigms and facilities for end-user data analysis. Modern cyberinfrastructure tools provide a diverse set of services that can be…
The coffea framework provides a new approach to High-Energy Physics analysis, via columnar operations, that improves time-to-insight, scalability, portability, and reproducibility of analysis. It is implemented with the Python programming…
Prior to the public release of Kubernetes it was difficult to conduct joint development of elaborate analysis facilities due to the highly non-homogeneous nature of hardware and network topology across compute facilities. However, since the…
In this paper we describe the architecture of a Platform as a Service (PaaS) oriented to computing and data analysis. In order to clarify the choices we made, we explain the features using practical examples, applied to several known usage…
This Letter considers the design for computing facilities that are complementary to the leadership class High Performance Computing (HPC) facilities. This design envisions a future where funding agencies are allocating greater resources for…
As AI agents take on increasingly long-running tasks involving sophisticated planning and execution, there is a corresponding need for novel interaction designs that enable deeper human-agent collaboration. However, most prior works…
HEP-Frame is a new C++ package designed to efficiently perform analyses of data sets from a very large number of events, like those available at the Large Hadron Collider (LHC) at CERN, Geneva. It mainly targets high performance servers and…
At high energy physics experiments, processing billions of records of structured numerical data from collider events to a few statistical summaries is a common task. The data processing is typically more complex than standard query…
As a part of the IRIS-HEP "Analysis Grand Challenge" activities, the Coffea-casa AF team executed a "200 Gbps Challenge". One of the goals of this challenge was to provide a setup for execution of a test notebook-style analysis on the…
Computer architecture design space is vast and complex. Tools are needed to explore new ideas and gain insights quickly, with low efforts and at a desired accuracy. We propose Calipers, a criticality-based framework to model key…
Most state of the art exploratory data analysis frameworks fall into one of the two extremes: they either focus on the high-performance computational, or on the interactive and open-ended aspects of the analysis. Arkouda is a framework that…
Without significant changes to data organization, management, and access (DOMA), HEP experiments will find scientific output limited by how fast data can be accessed and digested by computational resources. In this white paper we discuss…
Analysis cyberinfrastructure refers to the combination of software and computer hardware used to support late-stage data analysis in High Energy Physics (HEP). For the purposes of this white paper, late-stage data analysis refers…
HEP Analysis Facility is a cluster designed and implemented in Scientific Linux Cern 5.5 to grant High Energy Physics researchers one place where they can go to undertake a particular task or to provide a parallel processing architecture in…
Experimental Particle Physics has been at the forefront of analyzing the worlds largest datasets for decades. The HEP community was the first to develop suitable software and computing tools for this task. In recent times, new toolkits and…
Monte Carlo event generators are central to high-energy physics analysis. However, workflows based on handwritten scripts can be difficult to reuse, modify, and reproduce when multiple Monte Carlo models, tune variations, run variations,…
Research, especially in the social sciences and humanities, is increasingly reliant on the application of data science methods to analyze large amounts of (often private) data. Secure data enclaves provide a solution for managing and…
Many problems in Computer Science can be framed as the computation of queries over sequences, or "streams" of data units called events. The field of Complex Event Processing (CEP) relates to the techniques and tools developed to efficiently…
Security analysis is an essential activity in security engineering to identify potential system vulnerabilities and achieve security requirements in the early design phases. Due to the increasing complexity of modern systems, traditional…
We describe the status of the HepData database system, following a major re-development in time for the advent of LHC data. The new HepData system benefits from use of modern database and programming language technologies, as well as a…