Related papers: D0 Data Handling Operational Experience
Modern large-scale data-farms consist of hundreds of thousands of storage devices that span distributed infrastructure. Devices used in modern data centers (such as controllers, links, SSD- and HDD-disks) can fail due to hardware as well as…
The co-location of multiple database instances on resource constrained edge nodes creates significant cache contention, where traditional schemes are inefficient and unstable under dynamic workloads. To address this, we present SAM(a…
A longitudinal beam properties, data acquisition system has been commissioned to operate in the Fermilab booster ring. This system captures real time information including beam synchronous phase, bunch length, and coupled bunch instability…
The Real-Time Systems Engineering Department of the Scientific Computing Division at Fermilab has deployed set of customizations to our Off-The-Shelf Data Acquisition solution (otsdaq) at the Fermilab Test Beam Facility (FTBF) to read out…
In this paper, we propose an approach to the distributed storage and fusion of data for collective perception in resource-limited robot swarms. We demonstrate our approach in a distributed semantic classification scenario. We consider a…
This paper describes the design, fabrication, installation and performance of the new inner layer called Layer 0 (L0) that was inserted in the existing Run IIa Silicon Micro-Strip Tracker (SMT) of the D0 experiment at the Fermilab Tevatron…
Large systems biology projects can encompass several workgroups often located in different countries. An overview about existing data standards in systems biology and the management, storage, exchange and integration of the generated data…
A VME-based data acquisition system for beam-loss monitors has been developed and is in use in the Tevatron and Main Injector accelerators at the Fermilab complex. The need for enhanced beam-loss protection when the Tevatron is operating in…
Fermilab is the first High Energy Physics institution to transition from X.509 user certificates to authentication tokens in production systems. All the experiments that Fermilab hosts are now using JSON Web Token (JWT) access tokens in…
The Apache Accumulo database is an open source relaxed consistency database that is widely used for government applications. Accumulo is designed to deliver high performance on unstructured data such as graphs of network data. This paper…
The SiliconMicro-strip Tracker (SMT) at the D{\O} experiment in the Fermilab Tevatron collider has been operating since 2001. In 2006, an additional layer, referred to as 'Layer 0', was installed to improve impact parameter resolution and…
In recent IoT (Internet of Things) and Web 2.0 technologies, a critical problem arises with respect to storing and processing the large amount of collected data. In this paper we develop and evaluate distributed infrastructures for storing…
Fermilab is transitioning authentication and authorization for grid operations to using bearer tokens based on the WLCG Common JWT (JSON Web Token) Profile. One of the functionalities that Fermilab experimenters rely on is the ability to…
The Data Handling Pipeline ("Pipeline") has been developed for the Fermi Gamma-Ray Space Telescope (Fermi) Large Area Telescope (LAT) which launched in June 2008. Since then it has been in use to completely automate the production of data…
A distributed data warehouse system is one of the actual issues in the field of astroparticle physics. Famous experiments, such as TAIGA, KASCADE-Grande, produce tens of terabytes of data measured by their instruments. It is critical to…
Recent results suggest that very large datasets of teleoperated robot demonstrations can be used to train transformer-based models that have the potential to generalize to new scenes, robots, and tasks. However, curating, distributing, and…
The D0 experiment enjoyed a very successful data-collection run at the Fermilab Tevatron collider between 1992 and 1996. Since then, the detector has been upgraded to take advantage of improvements to the Tevatron and to enhance its physics…
In-memory key-value stores provide consistent low-latency access to all objects which is important for interactive large-scale applications like social media networks or online graph analytics and also opens up new application areas. But,…
We describe the offline computing system of the Belle experiment, consisting of a computing farm with one thousand IA-32 CPUs. Up to now, the Belle experiment has accumulated more than 120 fb$^{-1}$ of data, which is the world largest…
The XRootD system is used to transfer, store, and cache large datasets from high-energy physics (HEP). In this study we focus on its capability as distributed on-demand storage cache. Through exploring a large set of daily log files between…