English

Prompting as Scientific Inquiry

Computation and Language 2025-07-08 v2

Abstract

Prompting is the primary method by which we study and control large language models. It is also one of the most powerful: nearly every major capability attributed to LLMs-few-shot learning, chain-of-thought, constitutional AI-was first unlocked through prompting. Yet prompting is rarely treated as science and is frequently frowned upon as alchemy. We argue that this is a category error. If we treat LLMs as a new kind of complex and opaque organism that is trained rather than programmed, then prompting is not a workaround: it is behavioral science. Mechanistic interpretability peers into the neural substrate, prompting probes the model in its native interface: language. We contend that prompting is not inferior, but rather a key component in the science of LLMs.

Keywords

Cite

@article{arxiv.2507.00163,
  title  = {Prompting as Scientific Inquiry},
  author = {Ari Holtzman and Chenhao Tan},
  journal= {arXiv preprint arXiv:2507.00163},
  year   = {2025}
}
R2 v1 2026-07-01T03:40:20.884Z