RobotPerf: An Open-Source, Vendor-Agnostic, Benchmarking Suite for Evaluating Robotics Computing System Performance
Abstract
We introduce RobotPerf, a vendor-agnostic benchmarking suite designed to evaluate robotics computing performance across a diverse range of hardware platforms using ROS 2 as its common baseline. The suite encompasses ROS 2 packages covering the full robotics pipeline and integrates two distinct benchmarking approaches: black-box testing, which measures performance by eliminating upper layers and replacing them with a test application, and grey-box testing, an application-specific measure that observes internal system states with minimal interference. Our benchmarking framework provides ready-to-use tools and is easily adaptable for the assessment of custom ROS 2 computational graphs. Drawing from the knowledge of leading robot architects and system architecture experts, RobotPerf establishes a standardized approach to robotics benchmarking. As an open-source initiative, RobotPerf remains committed to evolving with community input to advance the future of hardware-accelerated robotics.
Keywords
Cite
@article{arxiv.2309.09212,
title = {RobotPerf: An Open-Source, Vendor-Agnostic, Benchmarking Suite for Evaluating Robotics Computing System Performance},
author = {Víctor Mayoral-Vilches and Jason Jabbour and Yu-Shun Hsiao and Zishen Wan and Martiño Crespo-Álvarez and Matthew Stewart and Juan Manuel Reina-Muñoz and Prateek Nagras and Gaurav Vikhe and Mohammad Bakhshalipour and Martin Pinzger and Stefan Rass and Smruti Panigrahi and Giulio Corradi and Niladri Roy and Phillip B. Gibbons and Sabrina M. Neuman and Brian Plancher and Vijay Janapa Reddi},
journal= {arXiv preprint arXiv:2309.09212},
year = {2024}
}