We present VoxelPrompt, an end-to-end image analysis agent that tackles free-form radiological tasks. Given any number of volumetric medical images and a natural language prompt, VoxelPrompt integrates a language model that generates executable code to invoke a jointly-trained, adaptable vision network. This code further carries out analytical steps to address practical quantitative aims, such as measuring the growth of a tumor across visits. The pipelines generated by VoxelPrompt automate analyses that currently require practitioners to painstakingly combine multiple specialized vision and statistical tools. We evaluate VoxelPrompt using diverse neuroimaging tasks and show that it can delineate hundreds of anatomical and pathological features, measure complex morphological properties, and perform open-language analysis of lesion characteristics. VoxelPrompt performs these objectives with an accuracy similar to that of specialist single-task models for image analysis, while facilitating a broad range of compositional biomedical workflows.
@article{arxiv.2410.08397,
title = {VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysis},
author = {Andrew Hoopes and Neel Dey and Victor Ion Butoi and John V. Guttag and Adrian V. Dalca},
journal= {arXiv preprint arXiv:2410.08397},
year = {2025}
}
Comments
22 pages, vision-language agent, medical image analysis, neuroimage foundation model