Reinforcement learning guided fuzz testing for a browser's HTML rendering engine
Artificial Intelligence
2023-07-28 v1 Software Engineering
Abstract
Generation-based fuzz testing can uncover various bugs and security vulnerabilities. However, compared to mutation-based fuzz testing, it takes much longer to develop a well-balanced generator that produces good test cases and decides where to break the underlying structure to exercise new code paths. We propose a novel approach to combine a trained test case generator deep learning model with a double deep Q-network (DDQN) for the first time. The DDQN guides test case creation based on a code coverage signal. Our approach improves the code coverage performance of the underlying generator model by up to 18.5\% for the Firefox HTML rendering engine compared to the baseline grammar based fuzzer.
Keywords
Cite
@article{arxiv.2307.14556,
title = {Reinforcement learning guided fuzz testing for a browser's HTML rendering engine},
author = {Martin Sablotny and Bjørn Sand Jensen and Jeremy Singer},
journal= {arXiv preprint arXiv:2307.14556},
year = {2023}
}