We introduce QSTN, an open-source Python framework for systematically generating responses from questionnaire-style prompts to support in-silico surveys and annotation tasks with large language models (LLMs). QSTN enables robust evaluation of questionnaire presentation, prompt perturbations, and response generation methods. Our extensive evaluation (>40 million survey responses) shows that question structure and response generation methods have a significant impact on the alignment of generated survey responses with human answers. We also find that answers can be obtained for a fraction of the compute cost, by changing the presentation method. In addition, we offer a no-code user interface that allows researchers to set up robust experiments with LLMs \emph{without coding knowledge}. We hope that QSTN will support the reproducibility and reliability of LLM-based research in the future.
@article{arxiv.2512.08646,
title = {QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Models},
author = {Maximilian Kreutner and Jens Rupprecht and Georg Ahnert and Ahmed Salem and Markus Strohmaier},
journal= {arXiv preprint arXiv:2512.08646},
year = {2026}
}
Comments
Accepted at 2026 EACL System Demonstrations The Python package is available at https://github.com/dess-mannheim/QSTN/