Metamorphic Testing of a Deep Learning based Forecaster
Abstract
In this paper, we present the Metamorphic Testing of an in-use deep learning based forecasting application. The application looks at the past data of system characteristics (e.g. `memory allocation') to predict outages in the future. We focus on two statistical / machine learning based components - a) detection of co-relation between system characteristics and b) estimating the future value of a system characteristic using an LSTM (a deep learning architecture). In total, 19 Metamorphic Relations have been developed and we provide proofs & algorithms where applicable. We evaluated our method through two settings. In the first, we executed the relations on the actual application and uncovered 8 issues not known before. Second, we generated hypothetical bugs, through Mutation Testing, on a reference implementation of the LSTM based forecaster and found that 65.9% of the bugs were caught through the relations.
Cite
@article{arxiv.1907.06632,
title = {Metamorphic Testing of a Deep Learning based Forecaster},
author = {Anurag Dwarakanath and Manish Ahuja and Sanjay Podder and Silja Vinu and Arijit Naskar and Koushik MV},
journal= {arXiv preprint arXiv:1907.06632},
year = {2019}
}
Comments
Paper published at the 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing (MET)