English

A Ceph S3 Object Data Store for HEP

Data Analysis, Statistics and Probability 2023-11-29 v1 High Energy Physics - Experiment
Authors: Nick Smith , Bo Jayatilaka , David Mason , Oliver Gutsche , Alison Peisker , Robert Illingworth , Chris Jones
View on arXiv ↗ PDF ↗

Abstract

We present a novel data format design that obviates the need for data tiers by storing individual event data products in column objects. The objects are stored and retrieved through Ceph S3 technology, with a layout designed to minimize metadata volume and maximize data processing parallelism. Performance benchmarks of data storage and retrieval are presented.

Cite

@article{arxiv.2311.16321,
  title  = {A Ceph S3 Object Data Store for HEP},
  author = {Nick Smith and Bo Jayatilaka and David Mason and Oliver Gutsche and Alison Peisker and Robert Illingworth and Chris Jones},
  journal= {arXiv preprint arXiv:2311.16321},
  year   = {2023}
}

Comments

CHEP2023 proceedings, to be published in EPJ Web of Conferences

Related papers

View all related →
Databases · Computer Science
Dynamic and Scalable Data Preparation for Object-Centric Process Mining
Lien Bosmans, Jari Peeperkorn, Alexandre Goossens, Giovanni Lugaresi +2
2024-10-02
High Energy Physics - Experiment · Physics
The Event as an Object-Relational Database: Avoiding the Dependency Nightmare
C. D. Jones
2007-05-23
Databases · Computer Science
Exploring Object Stores for High-Energy Physics Data Storage
Javier López-Gómez, Jakob Blomer
2021-09-08
Distributed, Parallel, and Cluster Computing · Computer Science
EGEON: Software-Defined Data Protection for Object Storage
Raul Saiz-Laudo, Marc Sanchez-Artigas
2022-06-28
Data Analysis, Statistics and Probability · Physics
10 Years of Object-Oriented Analysis on H1
Paul Laycock
2015-06-04
Databases · Computer Science
Column-Oriented Storage Techniques for MapReduce
Avrilia Floratou, Jignesh Patel, Eugene Shekita, Sandeep Tata
2011-05-24
Distributed, Parallel, and Cluster Computing · Computer Science
Tiered Object Storage using Persistent Memory
Johnu George, Ramdoot Pydipaty, Xinyuan Huang, Amit Saha +3
2018-07-18
Distributed, Parallel, and Cluster Computing · Computer Science
Parallel Writing of Nested Data in Columnar Formats
Jonas Hahnfeld, Jakob Blomer, Thorsten Kollegger
2024-10-21
High Energy Physics - Experiment · Physics
HEP-Frame: an Efficient Tool for Big Data Applications at the LHC
A. Pereira, A. Onofre, A. Proenca
2023-03-10
Databases · Computer Science
Enhancing Data-Awareness of Object-Centric Event Logs
Alexandre Goossens, Johannes De Smedt, Jan Vanthienen, Wil van der Aalst
2022-12-07
Computational Physics · Physics
Enabling Object Storage via shims for Grid Middleware
Samuel Cadellin Skipsey, Shaun De Witt, Alastair Dewhurst, David Britton +2
2016-01-20
Distributed, Parallel, and Cluster Computing · Computer Science
Parallel Data Object Creation: Towards Scalable Metadata Management in High-Performance I/O Library
Youjia Li, Robert Latham, Robert Ross, Ankit Agrawal +2
2025-06-19
Distributed, Parallel, and Cluster Computing · Computer Science
Performance Comparison of DAOS and Lustre for Object Data Storage Approaches
Nicolau Manubens, Simon D. Smart, Tiago Quintino, Adrian Jackson
2022-11-18
Distributed, Parallel, and Cluster Computing · Computer Science
Toward real-time data query systems in HEP
Jim Pivarski, David Lange, Thanat Jatuphattharachat
2017-11-09
Neural and Evolutionary Computing · Computer Science
Event management for large scale event-driven digital hardware spiking neural networks
Louis-Charles Caron, \and Michiel D'Haene, \and Frédéric Mailhot, \and Benjamin Schrauwen +1
2013-04-03
High Energy Physics - Phenomenology · Physics
HepData and JetWeb: HEP data archiving and model validation
A. Buckley, J. M. Butterworth, J. Monk, E. Nurse +3
2007-05-23
Data Structures and Algorithms · Computer Science
Efficient processing of raster and vector data
Fernando Silva-Coira, José R. Paramá, Susana Ladra, Juan R. López +1
2020-04-07
Distributed, Parallel, and Cluster Computing · Computer Science
Mapping Datasets to Object Storage System
Xiaowei, Chu, Jeff LeFevre, Aldrin Montana +4
2020-07-06
Computer Vision and Pattern Recognition · Computer Science
SESS: Self-Ensembling Semi-Supervised 3D Object Detection
Na Zhao, Tat-Seng Chua, Gim Hee Lee
2021-03-18
Databases · Computer Science
OASIS: Object-based Analytics Storage for Intelligent SQL Query Offloading in Scientific Tabular Workloads
Soon Hwang, Junhyeok Park, Junghyun Ryu, Seonghoon Ahn +7
2025-09-03
Computer Vision and Pattern Recognition · Computer Science
A Hybrid Neuromorphic Object Tracking and Classification Framework for Real-time Systems
Andres Ussa, Chockalingam Senthil Rajen, Deepak Singla, Jyotibdha Acharya +3
2020-07-23
Distributed, Parallel, and Cluster Computing · Computer Science
BitUP: Efficient Bitmap Data Storage Solution For User Profile
Derong Tang, Hank Wang
2023-10-30
Databases · Computer Science
Evolution of the ROOT Tree I/O
Jakob Blomer, Philippe Canal, Axel Naumann, Danilo Piparo
2021-02-03
Computer Vision and Pattern Recognition · Computer Science
Unleashing the Temporal Potential of Stereo Event Cameras for Continuous-Time 3D Object Detection
Jae-Young Kang, Hoonhee Cho, Kuk-Jin Yoon
2025-08-05
Human-Computer Interaction · Computer Science
EventColumn: Integrating Event Sequences into Tabular Visualizations
Jakob Zethofer, Andreas Hinterreiter, Lukas Schiefermüller, Belgin Mutlu +1
2026-05-08
R2 v1 2026-06-28T13:33:25.745Z