Chart What I Say: Exploring Cross-Modality Prompt Alignment in AI-Assisted Chart Authoring
Abstract
Recent chart-authoring systems, such as Amazon Q in QuickSight and Copilot for Power BI, demonstrate an emergent focus on supporting natural language input to share meaningful insights from data through chart creation. Currently, chart-authoring systems tend to integrate voice input capabilities by relying on speech-to-text transcription, processing spoken and typed input similarly. However, cross-modality input comparisons in other interaction domains suggest that the structure of spoken and typed-in interactions could notably differ, reflecting variations in user expectations based on interface affordances. Thus, in this work, we compare spoken and typed instructions for chart creation. Findings suggest that while both text and voice instructions cover chart elements and element organization, voice descriptions have a variety of command formats, element characteristics, and complex linguistic features. Based on these findings, we developed guidelines for designing voice-based authoring-oriented systems and additional features that can be incorporated into existing text-based systems to support speech modality.
Cite
@article{arxiv.2404.05103,
title = {Chart What I Say: Exploring Cross-Modality Prompt Alignment in AI-Assisted Chart Authoring},
author = {Nazar Ponochevnyi and Anastasia Kuzminykh},
journal= {arXiv preprint arXiv:2404.05103},
year = {2024}
}
Comments
Will be published In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA 2024)