The rise in capability and ubiquity of generative artificial intelligence (AI) technologies has enabled its application to the field of Socially Interactive Agents (SIAs). Despite rising interest in modern AI-powered components used for real-time SIA research, substantial friction remains due to the absence of a standardized and universal SIA framework. To target this absence, we developed Estuary: a multimodal (text, audio, and soon video) framework which facilitates the development of low-latency, real-time SIAs. Estuary seeks to reduce repeat work between studies and to provide a flexible platform that can be run entirely off-cloud to maximize configurability, controllability, reproducibility of studies, and speed of agent response times. We are able to do this by constructing a robust multimodal framework which incorporates current and future components seamlessly into a modular and interoperable architecture.
@article{arxiv.2410.20116,
title = {Estuary: A Framework For Building Multimodal Low-Latency Real-Time Socially Interactive Agents},
author = {Spencer Lin and Basem Rizk and Miru Jun and Andy Artze and Caitlin Sullivan and Sharon Mozgai and Scott Fisher},
journal= {arXiv preprint arXiv:2410.20116},
year = {2024}
}
Comments
To be published in ACM Intelligent Virtual Agents (IVA) 2024 [DOI: 10.1145/3652988.3696198] [ACM ISBN: 979-8-4007-0625-7/24/09]