English

A modular architecture for creating multimodal agents

Artificial Intelligence 2022-06-02 v1 Human-Computer Interaction Robotics

Abstract

The paper describes a flexible and modular platform to create multimodal interactive agents. The platform operates through an event-bus on which signals and interpretations are posted in a sequence in time. Different sensors and interpretation components can be integrated by defining their input and output as topics, which results in a logical workflow for further interpretations. We explain a broad range of components that have been developed so far and integrated into a range of interactive agents. We also explain how the actual interaction is recorded as multimodal data as well as in a so-called episodic Knowledge Graph. By analysing the recorded interaction, we can analyse and compare different agents and agent components.

Keywords

Cite

@article{arxiv.2206.00636,
  title  = {A modular architecture for creating multimodal agents},
  author = {Thomas Baier and Selene Baez Santamaria and Piek Vossen},
  journal= {arXiv preprint arXiv:2206.00636},
  year   = {2022}
}
R2 v1 2026-06-24T11:36:16.290Z