Data Engineering for Data Analytics: A Classification of the Issues, and Case Studies
Abstract
Consider the situation where a data analyst wishes to carry out an analysis on a given dataset. It is widely recognized that most of the analyst's time will be taken up with \emph{data engineering} tasks such as acquiring, understanding, cleaning and preparing the data. In this paper we provide a description and classification of such tasks into high-levels groups, namely data organization, data quality and feature engineering. We also make available four datasets and example analyses that exhibit a wide variety of these problems, to help encourage the development of tools and techniques to help reduce this burden and push forward research towards the automation or semi-automation of the data engineering process.
Cite
@article{arxiv.2004.12929,
title = {Data Engineering for Data Analytics: A Classification of the Issues, and Case Studies},
author = {Alfredo Nazabal and Christopher K. I. Williams and Giovanni Colavizza and Camila Rangel Smith and Angus Williams},
journal= {arXiv preprint arXiv:2004.12929},
year = {2020}
}
Comments
24 pages, 1 figure, submitted to IEEE Transactions on Knowledge and Data Engineering