Soft Measures for Extracting Causal Collective Intelligence
Abstract
Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP.
Cite
@article{arxiv.2409.18911,
title = {Soft Measures for Extracting Causal Collective Intelligence},
author = {Maryam Berijanian and Spencer Dork and Kuldeep Singh and Michael Riley Millikan and Ashlin Riggs and Aadarsh Swaminathan and Sarah L. Gibbs and Scott E. Friedman and Nathan Brugnone},
journal= {arXiv preprint arXiv:2409.18911},
year = {2025}
}
Comments
Camera-ready version accepted for publication in the EMNLP 2024 Workshop NLP4Science