English

Combining data and metadata: hybrid tabular file formats

Instrumentation and Methods for Astrophysics 2026-03-17 v1

Abstract

When working with astronomical data, metadata is also important. A general-purpose file format for transmission, processing and archiving large datasets should facilitate, among other things, both efficient processing of bulk data and encoding of rich semantic metadata. When choosing a format for a particular purpose sometimes no existing format satisfies both these requirements adequately, but combining one data-efficient and one metadata-rich format can be made to do so. This paper discusses considerations for designing such hybrid data/metadata formats, and reviews some examples such as VOParquet, FITS-plus and ECSV. We focus on tabular data, but some of the considerations may apply to other datatypes such as arrays as well.

Keywords

Cite

@article{arxiv.2603.15104,
  title  = {Combining data and metadata: hybrid tabular file formats},
  author = {Mark Taylor},
  journal= {arXiv preprint arXiv:2603.15104},
  year   = {2026}
}

Comments

4 pages, to appear in proceedings of ADASS XXXV

R2 v1 2026-07-01T11:22:01.739Z