Conclusion Stability for Natural Language Based Mining of Design Discussions
Abstract
Developer discussions range from in-person hallway chats to comment chains on bug reports. Being able to identify discussions that touch on software design would be helpful in documentation and refactoring software. Design mining is the application of machine learning techniques to correctly label a given discussion artifact, such as a pull request, as pertaining (or not) to design. In this paper we demonstrate a simple example of how design mining works. We then show how conclusion stability is poor on different artifact types and different projects. We show two techniques -- augmentation and context specificity -- that greatly improve the conclusion stability and cross-project relevance of design mining. Our new approach achieves AUC of 0.88 on within dataset classification and 0.80 on the cross-dataset classification task.
Cite
@article{arxiv.2106.09844,
title = {Conclusion Stability for Natural Language Based Mining of Design Discussions},
author = {Alvi Mahadi and Neil A. Ernst and Karan Tongay},
journal= {arXiv preprint arXiv:2106.09844},
year = {2021}
}
Comments
Accepted at Empirical Software Engineering Journal. Replication package at https://doi.org/10.5281/zenodo.4010208 and https://doi.org/10.5281/zenodo.4885980. arXiv admin note: substantial text overlap with arXiv:2001.01424