The rapid deployment of Large Language Models (LLMs) requires careful consideration of their effect on cybersecurity. Our work aims to improve the selection process of LLMs that are suitable for facilitating Secure Coding (SC). This raises challenging research questions, such as (RQ1) Which functionality can streamline the LLM evaluation? (RQ2) What should the evaluation measure? (RQ3) How to attest that the evaluation process is impartial? To address these questions, we introduce LLMSecCode, an open-source evaluation framework designed to assess LLM SC capabilities objectively. We validate the LLMSecCode implementation through experiments. When varying parameters and prompts, we find a 10% and 9% difference in performance, respectively. We also compare some results to reliable external actors, where our results show a 5% difference. We strive to ensure the ease of use of our open-source framework and encourage further development by external actors. With LLMSecCode, we hope to encourage the standardization and benchmarking of LLMs' capabilities in security-oriented code and tasks.
@article{arxiv.2408.16100,
title = {LLMSecCode: Evaluating Large Language Models for Secure Coding},
author = {Anton Rydén and Erik Näslund and Elad Michael Schiller and Magnus Almgren},
journal= {arXiv preprint arXiv:2408.16100},
year = {2024}
}
Comments
This manuscript serves as a complementary technical report to the proceedings version, which will be presented at the International Symposium on Cyber Security, Cryptography, and Machine Learning (CSCML) 2024