Related papers: Towards Next Generation Data Engineering Pipelines
Data mesh is an emerging decentralized approach to managing and generating value from analytical enterprise data at scale. It shifts the ownership of the data to the business domains closest to the data, promotes sharing and managing data…
Practitioners are increasingly turning to Extract-Load-Transform (ELT) pipelines with the widespread adoption of cloud data warehouses. However, designing these pipelines often involves significant manual work to ensure correctness. Recent…
Automated Machine Learning (AutoML) has been used successfully in settings where the learning task is assumed to be static. In many real-world scenarios, however, the data distribution will evolve over time, and it is yet to be shown…
In the era of Industry 4.0, system reliability engineering faces both challenges and opportunities. On the one hand, the complexity of cyber-physical systems, the integration of novel numerical technologies, and the handling of large…
Data stream processing systems (DSPSs) enable users to express and run stream applications to continuously process data streams. To achieve real-time data analytics, recent researches keep focusing on optimizing the system latency and…
Software organizations are increasingly incorporating machine learning (ML) into their product offerings, driving a need for new data management tools. Many of these tools facilitate the initial development of ML applications, but…
This paper provides a detailed survey of synthetic data techniques. We first discuss the expected goals of using synthetic data in data augmentation, which can be divided into four parts: 1) Improving Diversity, 2) Data Balancing, 3)…
The digital transformation of production requires new methods of data integration and storage, as well as decision making and support systems that work vertically and horizontally throughout the development, production, and use cycle. In…
As data continues to grow in scale and complexity, preparing, transforming, and analyzing it remains labor-intensive, repetitive, and difficult to scale. Since data contains knowledge and AI learns knowledge from it, the alignment between…
Deep learning is pervasive in our daily life, including self-driving cars, virtual assistants, social network services, healthcare services, face recognition, etc. However, deep neural networks demand substantial compute resources during…
In High Energy Physics (HEP), analysis metadata comes in many forms -- from theoretical cross-sections, to calibration corrections, to details about file processing. Correctly applying metadata is a crucial and often time-consuming step in…
The paper is devoted to studying the performance of a computational pipeline, the number of simultaneously executing stages of which at each time is bounded from above by a fixed number. A look at the restriction as a structural hazard…
Energy efficiency and reliability are vital design requirements of recent industrial networking solutions. Increased energy consumption, poor data access rates and unpredictable end-to-end data access latencies are catastrophic when…
The degradation of sewer pipes poses significant economical, environmental and health concerns. The maintenance of such assets requires structured plans to perform inspections, which are more efficient when structural and environmental…
Compiler optimization relies on sequences of passes to improve program performance. Selecting and ordering these passes automatically, known as compiler auto-tuning, is challenging due to the large and complex search space. Existing…
The next generation of high-power lasers enables repetition of experiments at orders of magnitude higher frequency than was possible using the prior generation. Facilities requiring human intervention between laser repetitions need to adapt…
A growing number of applications that generate massive streams of data need intelligent data processing and online analysis. Real-time surveillance systems, telecommunication systems, sensor networks and other dynamic environments are such…
Optimized data transfer services are highly demanded nowadays, due to the large amounts of data which are frequently being produced and accessed. In this paper we consider several data transfer service optimization problems (optimal server…
By programming both the data plane and the control plane, network operators can customize their networks based on their needs, regardless of the hardware manufacturer. Control plane programming, a major component of the SDN (Software…
Building high-quality knowledge graphs (KGs) from diverse sources requires combining methods for information extraction, data transformation, ontology mapping, entity matching, and data fusion. Numerous methods and tools exist for each of…