This paper introduces BioVL-QR, a biochemical vision-and-language dataset comprising 23 egocentric experiment videos, corresponding protocols, and vision-and-language alignments. A major challenge in understanding biochemical videos is detecting equipment, reagents, and containers because of the cluttered environment and indistinguishable objects. Previous studies assumed manual object annotation, which is costly and time-consuming. To address the issue, we focus on Micro QR Codes. However, detecting objects using only Micro QR Codes is still difficult due to blur and occlusion caused by object manipulation. To overcome this, we propose an object labeling method combining a Micro QR Code detector with an off-the-shelf hand object detector. As an application of the method and BioVL-QR, we tackled the task of localizing the procedural steps in an instructional video. The experimental results show that using Micro QR Codes and our method improves biochemical video understanding. Data and code are available through https://nishi10mo.github.io/BioVL-QR/
@article{arxiv.2404.03161,
title = {BioVL-QR: Egocentric Biochemical Vision-and-Language Dataset Using Micro QR Codes},
author = {Tomohiro Nishimoto and Taichi Nishimura and Koki Yamamoto and Keisuke Shirai and Hirotaka Kameko and Yuto Haneji and Tomoya Yoshida and Keiya Kajimura and Taiyu Cui and Chihiro Nishiwaki and Eriko Daikoku and Natsuko Okuda and Fumihito Ono and Shinsuke Mori},
journal= {arXiv preprint arXiv:2404.03161},
year = {2025}
}