Evaluating SAP Joule for Code Generation

Joshua Heisler; Johannes Reisinger; Andreas Fischer

Evaluating SAP Joule for Code Generation

Software Engineering 2025-09-30 v1 Artificial Intelligence

Authors: Joshua Heisler , Johannes Reisinger , Andreas Fischer

Abstract

SAP has released its own proprietary generative model SAP Joule, intended for various generative tasks, including serving as a code assistant for software engineers. While Joule is yet not focused on SAP-specific ABAP code generation, it can be used for other common languages, including Javascript. This paper compares SAP Joules Javascript coding capabilities against a total of 29 other models using the HumanEval-X Javascript benchmark. SAP Joule achieves a strict accuracy of 80.49% as the fifth best model in our evaluation. To the best of our knowledge, this is the first comparative evaluation of SAP Joule code generation capabilities.

Cite

@article{arxiv.2509.24828,
  title  = {Evaluating SAP Joule for Code Generation},
  author = {Joshua Heisler and Johannes Reisinger and Andreas Fischer},
  journal= {arXiv preprint arXiv:2509.24828},
  year   = {2025}
}

Related papers

View all related →