We present Umwelt, an authoring environment for interactive multimodal data representations. In contrast to prior approaches, which center the visual modality, Umwelt treats visualization, sonification, and textual description as coequal representations: they are all derived from a shared abstract data model, such that no modality is prioritized over the others. To simplify specification, Umwelt evaluates a set of heuristics to generate default multimodal representations that express a dataset's functional relationships. To support smoothly moving between representations, Umwelt maintains a shared query predicated that is reified across all modalities -- for instance, navigating the textual description also highlights the visualization and filters the sonification. In a study with 5 blind / low-vision expert users, we found that Umwelt's multimodal representations afforded complementary overview and detailed perspectives on a dataset, allowing participants to fluidly shift between task- and representation-oriented ways of thinking.
@article{arxiv.2403.00106,
title = {Umwelt: Accessible Structured Editing of Multimodal Data Representations},
author = {Jonathan Zong and Isabella Pedraza Pineros and Mengzhu Katie Chen and Daniel Hajas and Arvind Satyanarayan},
journal= {arXiv preprint arXiv:2403.00106},
year = {2024}
}