Hypothesis formulation and testing are central to empirical research. A strong hypothesis is a best guess based on existing evidence and informed by a comprehensive view of relevant literature. However, with exponential increase in the number of scientific articles published annually, manual aggregation and synthesis of evidence related to a given hypothesis is a challenge. Our work explores the ability of current large language models (LLMs) to discern evidence in support or refute of specific hypotheses based on the text of scientific abstracts. We share a novel dataset for the task of scientific hypothesis evidencing using community-driven annotations of studies in the social sciences. We compare the performance of LLMs to several state-of-the-art benchmarks and highlight opportunities for future research in this area. The dataset is available at https://github.com/Sai90000/ScientificHypothesisEvidencing.git
@article{arxiv.2309.06578,
title = {Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciences},
author = {Sai Koneru and Jian Wu and Sarah Rajtmajer},
journal= {arXiv preprint arXiv:2309.06578},
year = {2024}
}