MARRS: Multimodal Reference Resolution System
Abstract
Successfully handling context is essential for any dialog understanding task. This context maybe be conversational (relying on previous user queries or system responses), visual (relying on what the user sees, for example, on their screen), or background (based on signals such as a ringing alarm or playing music). In this work, we present an overview of MARRS, or Multimodal Reference Resolution System, an on-device framework within a Natural Language Understanding system, responsible for handling conversational, visual and background context. In particular, we present different machine learning models to enable handing contextual queries; specifically, one to enable reference resolution, and one to handle context via query rewriting. We also describe how these models complement each other to form a unified, coherent, lightweight system that can understand context while preserving user privacy.
Cite
@article{arxiv.2311.01650,
title = {MARRS: Multimodal Reference Resolution System},
author = {Halim Cagri Ates and Shruti Bhargava and Site Li and Jiarui Lu and Siddhardha Maddula and Joel Ruben Antony Moniz and Anil Kumar Nalamalapu and Roman Hoang Nguyen and Melis Ozyildirim and Alkesh Patel and Dhivya Piraviperumal and Vincent Renkens and Ankit Samal and Thy Tran and Bo-Hsiang Tseng and Hong Yu and Yuan Zhang and Rong Zou},
journal= {arXiv preprint arXiv:2311.01650},
year = {2023}
}
Comments
Sixth Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC 2023)