This technical report describes the CONE approach for Ego4D Natural Language Queries (NLQ) Challenge in ECCV 2022. We leverage our model CONE, an efficient window-centric COarse-to-fiNE alignment framework. Specifically, CONE dynamically slices the long video into candidate windows via a sliding window approach. Centering at windows, CONE (1) learns the inter-window (coarse-grained) semantic variance through contrastive learning and speeds up inference by pre-filtering the candidate windows relevant to the NL query, and (2) conducts intra-window (fine-grained) candidate moments ranking utilizing the powerful multi-modal alignment ability of the contrastive vision-text pre-trained model EgoVLP. On the blind test set, CONE achieves 15.26 and 9.24 for R1@IoU=0.3 and R1@IoU=0.5, respectively.
Cite
@article{arxiv.2211.08776,
title = {An Efficient COarse-to-fiNE Alignment Framework @ Ego4D Natural Language Queries Challenge 2022},
author = {Zhijian Hou and Wanjun Zhong and Lei Ji and Difei Gao and Kun Yan and Wing-Kwong Chan and Chong-Wah Ngo and Zheng Shou and Nan Duan},
journal= {arXiv preprint arXiv:2211.08776},
year = {2022}
}
Comments
Technical report for ECCV 2022 Ego4D workshop, 4 pages, 2 figures, 2 tables. arXiv admin note: substantial text overlap with arXiv:2209.10918