Given multiple videos of the same task, procedure learning addresses identifying the key-steps and determining their order to perform the task. For this purpose, existing approaches use the signal generated from a pair of videos. This makes key-steps discovery challenging as the algorithms lack inter-videos perspective. Instead, we propose an unsupervised Graph-based Procedure Learning (GPL) framework. GPL consists of the novel UnityGraph that represents all the videos of a task as a graph to obtain both intra-video and inter-videos context. Further, to obtain similar embeddings for the same key-steps, the embeddings of UnityGraph are updated in an unsupervised manner using the Node2Vec algorithm. Finally, to identify the key-steps, we cluster the embeddings using KMeans. We test GPL on benchmark ProceL, CrossTask, and EgoProceL datasets and achieve an average improvement of 2% on third-person datasets and 3.6% on EgoProceL over the state-of-the-art.
@article{arxiv.2311.03550,
title = {United We Stand, Divided We Fall: UnityGraph for Unsupervised Procedure Learning from Videos},
author = {Siddhant Bansal and Chetan Arora and C. V. Jawahar},
journal= {arXiv preprint arXiv:2311.03550},
year = {2023}
}
Comments
13 pages, 6 figures, Accepted in Winter Conference on Applications of Computer Vision (WACV), 2024