Affordance-centric Question-driven Task Completion (AQTC) for Egocentric Assistant introduces a groundbreaking scenario. In this scenario, through learning instructional videos, AI assistants provide users with step-by-step guidance on operating devices. In this paper, we present a solution for enhancing video alignment to improve multi-step inference. Specifically, we first utilize VideoCLIP to generate video-script alignment features. Afterwards, we ground the question-relevant content in instructional videos. Then, we reweight the multimodal context to emphasize prominent features. Finally, we adopt GRU to conduct multi-step inference. Through comprehensive experiments, we demonstrate the effectiveness and superiority of our method, which secured the 2nd place in CVPR'2023 AQTC challenge. Our code is available at https://github.com/zcfinal/LOVEU-CVPR23-AQTC.
@article{arxiv.2306.14412,
title = {A Solution to CVPR'2023 AQTC Challenge: Video Alignment for Multi-Step Inference},
author = {Chao Zhang and Shiwei Wu and Sirui Zhao and Tong Xu and Enhong Chen},
journal= {arXiv preprint arXiv:2306.14412},
year = {2023}
}
Comments
5 pages, 1 figure, technical report for track3 of CVPR 2023 LOVEU challenge