We characterize and demonstrate how the principles of direct manipulation can improve interaction with large language models. This includes: continuous representation of generated objects of interest; reuse of prompt syntax in a toolbar of commands; manipulable outputs to compose or control the effect of prompts; and undo mechanisms. This idea is exemplified in DirectGPT, a user interface layer on top of ChatGPT that works by transforming direct manipulation actions to engineered prompts. A study shows participants were 50% faster and relied on 50% fewer and 72% shorter prompts to edit text, code, and vector images compared to baseline ChatGPT. Our work contributes a validated approach to integrate LLMs into traditional software using direct manipulation. Data, code, and demo available at https://osf.io/3wt6s.
@article{arxiv.2310.03691,
title = {DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models},
author = {Damien Masson and Sylvain Malacria and Géry Casiez and Daniel Vogel},
journal= {arXiv preprint arXiv:2310.03691},
year = {2025}
}
Comments
In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI '24)