Towards Interactively Improving ML Data Preparation Code via "Shadow Pipelines"
Abstract
Data scientists develop ML pipelines in an iterative manner: they repeatedly screen a pipeline for potential issues, debug it, and then revise and improve its code according to their findings. However, this manual process is tedious and error-prone. Therefore, we propose to support data scientists during this development cycle with automatically derived interactive suggestions for pipeline improvements. We discuss our vision to generate these suggestions with so-called shadow pipelines, hidden variants of the original pipeline that modify it to auto-detect potential issues, try out modifications for improvements, and suggest and explain these modifications to the user. We envision to apply incremental view maintenance-based optimisations to ensure low-latency computation and maintenance of the shadow pipelines. We conduct preliminary experiments to showcase the feasibility of our envisioned approach and the potential benefits of our proposed optimisations.
Cite
@article{arxiv.2404.19591,
title = {Towards Interactively Improving ML Data Preparation Code via "Shadow Pipelines"},
author = {Stefan Grafberger and Paul Groth and Sebastian Schelter},
journal= {arXiv preprint arXiv:2404.19591},
year = {2024}
}