Related papers: Data engineering for archive evolution
The hardware for data archiving has expanded capacities for digital storage enormously in the past decade or more. This article charts the efforts of IUCr to facilitate discussions and plans relating to raw data archiving and reuse within…
Broad efforts are underway to capture metadata about research software and retain it across services; notable in this regard is the CodeMeta project. What metadata are important to have about (research) software? What metadata are useful…
Astronomy produces extremely large data sets from ground-based telescopes, space missions, and simulation. The volume and complexity of these rich data sets require new approaches and advanced tools to understand the information contained…
Evolutionary change over time in the context of data pipelines is certain, especially with regard to the structure and semantics of data as well as to the pipeline operators. Dealing with these changes, i.e. providing long-term maintenance,…
Data analysis in space sciences has been performed exclusively visually for years, despite the fact that the largest amount of data belongs to non-visible portions of the electromagnetic spectrum. This, on the one hand, limits the study of…
As the volume of data grows, astronomers are increasingly faced with choices on what data to keep -- and what to throw away. Recent work evaluating the JPEG2000 (ISO/IEC 15444) standards as a future data format standard in astronomy has…
The ability to collect and analyze large amounts of data is a growing problem within the scientific community. The growing gap between data and users calls for innovative tools that address the challenges faced by big data volume, velocity…
Here I present an investigation on the evolution and use of vocabulary in data science in the last 13 years. Based on a rigorous statistical analysis, a database with 12,787 documents containing the words "data science" in the title,…
Astronomy is undergoing through a methodological revolution triggered by an unprecedented wealth of complex and accurate data. The new panchromatic, synoptic sky surveys require advanced tools for discovering patterns and trends hidden…
The nature of scientific and technological data collection is evolving rapidly: data volumes and rates grow exponentially, with increasing complexity and information content, and there has been a transition from static data sets to data…
In materials sciences, a large amount of research data is generated through a broad spectrum of different experiments. As of today, experimental research data including meta-data in materials science is often stored decentralized by the…
A persistent challenge in astronomical machine learning is a systematic bias where predictions compress the dynamic range of true values-high values are consistently predicted too low while low values are predicted too high. Understanding…
We have derived the uncertainties to be expected in the derivation of galaxy physical properties (star formation history, age, metallicity, reddening) when comparing broad-band photometry to the predictions of evolutionary synthesis models.…
This white paper offers cautionary observations about the planning and development of new, large radio astronomy instruments. Complexity is a strong cost driver so every effort should be made to assign differing science requirements to…
Digital Engineering, the digital transformation of engineering to leverage digital technologies, is coming globally. This paper explores digital systems engineering, which aims at developing theory, methods, models, and tools to support the…
Data-Augmentation (DA) is known to improve performance across tasks and datasets. We propose a method to theoretically analyze the effect of DA and study questions such as: how many augmented samples are needed to correctly estimate the…
Astronomical software is now a fact of daily life for all hands-on members of our community. Purpose-built software for data reduction and modeling tasks becomes ever more critical as we handle larger amounts of data and simulations.…
Long-term data-driven studies have become indispensable in many areas of science. Often, the data formats, structures and semantics of data change over time, the data sets evolve. Therefore, studies over several decades in particular have…
Traditional science searched for new objects and phenomena that led to discoveries. Tomorrow's science will combine together the large pool of information in scientific archives and make discoveries. Scienthists are currently keen to…
In this paper, we tackle the incremental maintenance of Datalog inference materialisation when the rule set can be updated. This is particularly relevant in the context of the Internet of Things and Edge computing where smart devices may…