We introduce FinNLI, a benchmark dataset for Financial Natural Language Inference (FinNLI) across diverse financial texts like SEC Filings, Annual Reports, and Earnings Call transcripts. Our dataset framework ensures diverse premise-hypothesis pairs while minimizing spurious correlations. FinNLI comprises 21,304 pairs, including a high-quality test set of 3,304 instances annotated by finance experts. Evaluations show that domain shift significantly degrades general-domain NLI performance. The highest Macro F1 scores for pre-trained (PLMs) and large language models (LLMs) baselines are 74.57% and 78.62%, respectively, highlighting the dataset's difficulty. Surprisingly, instruction-tuned financial LLMs perform poorly, suggesting limited generalizability. FinNLI exposes weaknesses in current LLMs for financial reasoning, indicating room for improvement.
@article{arxiv.2504.16188,
title = {FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking},
author = {Jabez Magomere and Elena Kochkina and Samuel Mensah and Simerjot Kaur and Charese H. Smiley},
journal= {arXiv preprint arXiv:2504.16188},
year = {2025}
}