Specifications: The missing link to making the development of LLM systems an engineering discipline
Abstract
Despite the significant strides made by generative AI in just a few short years, its future progress is constrained by the challenge of building modular and robust systems. This capability has been a cornerstone of past technological revolutions, which relied on combining components to create increasingly sophisticated and reliable systems. Cars, airplanes, computers, and software consist of components-such as engines, wheels, CPUs, and libraries-that can be assembled, debugged, and replaced. A key tool for building such reliable and modular systems is specification: the precise description of the expected behavior, inputs, and outputs of each component. However, the generality of LLMs and the inherent ambiguity of natural language make defining specifications for LLM-based components (e.g., agents) both a challenging and urgent problem. In this paper, we discuss the progress the field has made so far-through advances like structured outputs, process supervision, and test-time compute-and outline several future directions for research to enable the development of modular and reliable LLM-based systems through improved specifications.
Cite
@article{arxiv.2412.05299,
title = {Specifications: The missing link to making the development of LLM systems an engineering discipline},
author = {Ion Stoica and Matei Zaharia and Joseph Gonzalez and Ken Goldberg and Koushik Sen and Hao Zhang and Anastasios Angelopoulos and Shishir G. Patil and Lingjiao Chen and Wei-Lin Chiang and Jared Q. Davis},
journal= {arXiv preprint arXiv:2412.05299},
year = {2024}
}