Related papers: Using script generators for pipeline prototyping
A data analysis pipeline is a structured sequence of steps that transforms raw data into meaningful insights by integrating various analysis algorithms. In this paper, we propose a novel statistical test to assess the significance of data…
Large language models are redefining software engineering by implementing AI-powered techniques throughout the whole software development process, including requirement gathering, software architecture, code generation, testing, and…
Deploying large language model-based code generation in real-world client-side development remains challenging due to heterogeneous inputs, strict engineering constraints, and complex interaction logic expressed in product requirement…
Forecasting transformative technologies remains a critical but challenging task, particularly in fast-evolving domains such as Information and Communication Technologies (ICTs). Traditional expert-based methods struggle to keep pace with…
Modular design is a key challenge for enabling large-scale reuse of hardware modules. Unlike software, however, hardware designs correspond to physical circuits and inherit constraints from them. Timing constraints -- which cycle a signal…
Processing high-throughput DNA sequencing data of individuals or populations requires stringing together independent software tools with many parameters, often leading to non-reproducible pipelines and datasets. We developed grenepipe to…
Data engineering pipelines are a widespread way to provide high-quality data for all kinds of data science applications. However, numerous challenges still remain in the composition and operation of such pipelines. Data engineering…
A data analysis pipeline is a structured sequence of steps that transforms raw data into meaningful insights by integrating multiple analysis algorithms. In many practical applications, analytical findings are obtained only after data pass…
Software development projects involve the use of a wide range of tools to produce a software artifact. Software repositories such as source control systems have become a focus for emergent research because they are a source of rich…
The Kieker observability framework is a tool that provides users with the means to design a custom observability pipeline for their application. Originally tailored for Java, supporting Python with Kieker is worthwhile. Python's popularity…
We present a novel computational paradigm for process design in manufacturing processes that incorporates simulation responses to optimize manufacturing process parameters in high-dimensional temporal and spatial design spaces. We developed…
Software is a critical aspect of large-scale science, providing essential capabilities for making scientific discoveries. Large-scale scientific projects are vast in scope, with lifespans measured in decades and costs exceeding hundreds of…
This paper argues for an accelerator development toolchain that takes into account the whole system containing the accelerator. With whole-system visibility, the toolchain can better assist accelerator scoping and composition in the context…
Deploying a Machine Learning (ML) training pipeline into production requires good software engineering practices. Unfortunately, the typical data science workflow often leads to code that lacks critical software quality attributes. This…
With the growing interest in deep learning algorithms and computational design in the architectural field, the need for large, accessible and diverse architectural datasets increases. We decided to tackle this problem by constructing a…
Pipelining is a design technique for logical circuits that allows for higher throughput than circuits in which multiple computations are fed through the system one after the other. It allows for much faster computation than architectures in…
A program that we call the QUEST Data Processing Software Pipeline has been written to process the large volumes of data produced by the QUEST camera on the Samuel Oschin Schmidt Telescope at the Palomar Observatory. The program carries out…
Quality control of assembly processes is essential in manufacturing to ensure not only the quality of individual components but also their proper integration into the final product. To assist in this matter, automated assembly control using…
Linting tools automatically identify source code fragments that do not follow a set of predefined standards. Such feedback tools are equally desirable for "linting" agile development processes. However, providing concrete feedback on…
Recent advances in large language models (LLMs) transform how machine learning (ML) pipelines are developed and evaluated. LLMs enable a new type of workload, agentic pipeline search, in which autonomous or semi-autonomous agents generate,…