Related papers: Big Data in HEP: A comprehensive use case study
Motivated by the expectation that new physics may manifest itself in the form of very heavy new particles, most of the operation time of the LHC is devoted to $pp$ collisions at the highest achievable energies and collision rates. The large…
One of the biggest challenges in the High-Luminosity LHC (HL- LHC) era will be the significantly increased data size to be recorded and analyzed from the collisions at the ATLAS and CMS experiments. ServiceX is a software R&D project in the…
Numerous challenges persist in High Energy Physics (HEP), the addressing of which requires advancements in detection technology, computational methods, data analysis frameworks, and phenomenological designs. We provide a concise yet…
The IRIS-HEP Analysis Grand Challenge (AGC) is designed to be a realistic environment for investigating how analysis methods scale to the demands of the HL-LHC. The analysis task is based on publicly available Open Data and allows for…
This document reports the sequence of practices and methodologies implemented during the Big Data course. It details the workflow beginning with the processing of the Epsilon dataset through group and individual strategies, followed by text…
Enterprises operate large data lakes using Hadoop and Spark frameworks that (1) run a plethora of tools to automate powerful data preparation/transformation pipelines, (2) run on shared, large clusters to (3) perform many different…
A second major upgrade of the LHCb experiment is necessary to allow full exploitation of the High Luminosity LHC for flavour physics. The new experiment will operate in Run 5 of the LHC at a luminosity up to $1.5\times…
We present CosmoHub (https://cosmohub.pic.es), a web application based on Hadoop to perform interactive exploration and distribution of massive cosmological datasets. Recent Cosmology seeks to unveil the nature of both dark matter and dark…
Most of the popular Big Data analytics tools evolved to adapt their working environment to extract valuable information from a vast amount of unstructured data. The ability of data mining techniques to filter this helpful information from…
The LHCb experiment at the Large Hadron Collider (LHC) is performing high precision measurements in the flavour sector. An excellent performance of the particle identification (PID) detectors as well as the development of new data taking…
Data analysis in fundamental sciences nowadays is an essential process that pushes frontiers of our knowledge and leads to new discoveries. At the same time we can see that complexity of those analyses increases fast due to a)~enormous…
The study group on data preservation in high energy physics, DPHEP, is moving to a new collaboration structure, which will focus on the implementation of preservation projects, such as those described in the group's large scale report…
A large hadron machine like the LHC with its high track multiplicities always asks for powerful tools that drastically reduce the large background while selecting signal events efficiently. Actually such tools are widely needed and used in…
In this new era of large data, it is important to make sure we do not miss any signs of new physics. Using the publicly-available open data collected by the arXiv.org experiment in the \texttt{hep-ph} channel, corresponding to a raw total…
Data from particle physics experiments are unique and are often the result of a very large investment of resources. Given the potential scientific impact of these data, which goes far beyond the immediate priorities of the experimental…
There is growing interest in the issues of preservation and re-use of the records of science, in the "digital era". The aim of the PARSE.Insight project, partly financed by the European Commission under the Seventh Framework Program, is…
The growing complexity and variety of Big Data platforms makes it both difficult and time consuming for all system users to properly setup and operate the systems. Another challenge is to compare the platforms in order to choose the most…
Signatures of new physics at the LHC are varied and, by nature, often very different from those of Standard Model processes. Novel experimental techniques, including dedicated data streams, are exploited to enhance the sensitivity of the…
Data processing frameworks are an essential part of HEP experiments' software stacks. Frameworks provide a means by which code developers can undertake the essential tasks of physics data processing, accessing relevant inputs and storing…
The searches for new particles that could be constituents of the dark matter in the universe are an essential part of the physics program of the experiments at the Large Hadron Collider. An overview of recent dark matter candidate searches…