OSS: Open Suturing Skills Vision-Based Assessment Challenge 2024-2025
Abstract
Achieving high levels of surgical skill through effective training is essential for optimal patient outcomes. Automated, data-driven skill assessment holds significant potential to improve surgical training. While machine learning-based methods are increasingly popular for assessing skills in minimally invasive surgery, their application to open surgery remains limited. We present the results of a dedicated MICCAI challenge designed to benchmark and advance vision-based skill assessment in open surgery. The challenge dataset comprises videos of an open suturing training task recorded with a static GoPro camera in a dry-lab setting, with instrument trajectories available in addition to the primary video modality. The OSS Challenge was hosted over two consecutive years, comprising two and three independent tasks, respectively: (1) classifying skill level into four classes, (2) predicting the full Objective Structured Assessment of Technical Skills across eight categories, and (3) tracking hands and surgical tools. Participants submitted diverse solutions including deep learning-based video models, tracking-driven methods, and hybrid approaches. General-purpose spatiotemporal video models consistently achieved the strongest performance, though conceptually diverse approaches reached competitive levels when well-executed. Predicting fine-grained OSATS scores remains challenging but benefits substantially from increased training data. Keypoint tracking proves difficult given frequent occlusions and out-of-frame instances, limiting current applicability for motion-based skill analysis. This work benchmarks innovative and diverse solutions for surgical skill assessment, highlighting both the promise and current limitations of video-based evaluation in open surgery and identifying critical directions for advancing automated skill assessment toward clinical impact.
Cite
@article{arxiv.2605.22200,
title = {OSS: Open Suturing Skills Vision-Based Assessment Challenge 2024-2025},
author = {Hanna Hoffmann and Setareh Bady and Claas de Boer and Max Kirchner and Jan Egger and Rainer Röhrig and Frank Hölzle and Lennart Johannes Gruber and Kunpeng Xie and Marlon Neuhaus and Victor Alves and Guilherme Barbosa and Leonardo Barroso and João Carvalho and Hao Chen and Gabriella d'Albenzio and André Ferreira and Nuno Gomes and Yuichiro Hayashi and Kousuke Hirasawa and Rebecca Hisey and Seungjae Hong and Seoi Jeong and Tiago Jesus and Daehong Kang and Satoshi Kasai and Shunsuke Kikuchi and Takayuki Kitasaka and Satoshi Kondo and Hyoun-Joong Kong and Youngbin Kong and Atsushi Kouno and Shlomi Laufer and Kyu Eun Lee and Bining Long and Nooshin Maghsoodi and Hiroki Matsuzaki and Evangelos Mazomenos and Ori Meiraz and Kensaku Mori and Marina Music and Masahiro Oda and Roi Papo and Jieun Park and Rafael Piexoto and Saeid Rezaei and Mariana Ribeiro and Soyeon Shin and Yang Shu and Idan Smoller and Danail Stoyanov and Yihui Wang and Xinkai Zhao and Sebastian Bodenstedt and Isabel Funke and Stefanie Speidel and Behrus Hinrichs-Puladi},
journal= {arXiv preprint arXiv:2605.22200},
year = {2026}
}
Comments
Stefanie Speidel and Behrus Hinrichs-Puladi jointly supervised this work. Submitted to MEDIA