ProIO is a new event-oriented streaming data format which utilizes Google's Protocol Buffers (protobuf) to be flexible and highly language-neutral. The ProIO concept is described here along with its software implementations. The performance of the ProIO concept for a dataset with Monte-Carlo event records used in high-energy physics was benchmarked and compared/contrasted with ROOT I/O. Various combinations of general-purpose compression and variable-length integer encoding available in protobuf were used to investigate the relationship between I/O performance and size-on-disk in a few key scenarios.
Cite
@article{arxiv.1812.03967,
title = {ProIO: An Event-Based I/O Stream Format for Protobuf Messages},
author = {D. Blyth and J. Alcaraz and S. Binet and S. V. Chekanov},
journal= {arXiv preprint arXiv:1812.03967},
year = {2019}
}