We introduce KFinEval-Pilot, a benchmark suite specifically designed to evaluate large language models (LLMs) in the Korean financial domain. Addressing the limitations of existing English-centric benchmarks, KFinEval-Pilot comprises over 1,000 curated questions across three critical areas: financial knowledge, legal reasoning, and financial toxicity. The benchmark is constructed through a semi-automated pipeline that combines GPT-4-generated prompts with expert validation to ensure domain relevance and factual accuracy. We evaluate a range of representative LLMs and observe notable performance differences across models, with trade-offs between task accuracy and output safety across different model families. These results highlight persistent challenges in applying LLMs to high-stakes financial applications, particularly in reasoning and safety. Grounded in real-world financial use cases and aligned with the Korean regulatory and linguistic context, KFinEval-Pilot serves as an early diagnostic tool for developing safer and more reliable financial AI systems.
@article{arxiv.2504.13216,
title = {KFinEval-Pilot: A Comprehensive Benchmark Suite for Korean Financial Language Understanding},
author = {Bokwang Hwang and Seonkyu Lim and Taewoong Kim and Yongjae Geun and Sunghyun Bang and Sohyun Park and Jihyun Park and Myeonggyu Lee and Jinwoo Lee and Yerin Kim and Jinsun Yoo and Jingyeong Hong and Jina Park and Yongchan Kim and Suhyun Kim and Younggyun Hahm and Yiseul Lee and Yejee Kang and Chanhyuk Yoon and Chansu Lee and Heeyewon Jeong and Jiyeon Lee and Seonhye Gu and Hyebin Kang and Yousang Cho and Hangyeol Yoo and KyungTae Lim},
journal= {arXiv preprint arXiv:2504.13216},
year = {2025}
}