Two-stream network-driven vision-based tactile sensor for object feature extraction and fusion perception
Abstract
Tactile perception is crucial for embodied intelligent robots to recognize objects. Vision-based tactile sensors extract object physical attributes multidimensionally using high spatial resolution; however, this process generates abundant redundant information. Furthermore, single-dimensional extraction, lacking effective fusion, fails to fully characterize object attributes. These challenges hinder the improvement of recognition accuracy. To address this issue, this study introduces a two-stream network feature extraction and fusion perception strategy for vision-based tactile systems. This strategy employs a distributed approach to extract internal and external object features. It obtains depth map information through three-dimensional reconstruction while simultaneously acquiring hardness information by measuring contact force data. After extracting features with a convolutional neural network (CNN), weighted fusion is applied to create a more informative and effective feature representation. In standard tests on objects of varying shapes and hardness, the force prediction error is 0.06 N (within a 12 N range). Hardness recognition accuracy reaches 98.0%, and shape recognition accuracy reaches 93.75%. With fusion algorithms, object recognition accuracy in actual grasping scenarios exceeds 98.5%. Focused on object physical attributes perception, this method enhances the artificial tactile system ability to transition from perception to cognition, enabling its use in embodied perception applications.
Cite
@article{arxiv.2510.12528,
title = {Two-stream network-driven vision-based tactile sensor for object feature extraction and fusion perception},
author = {Muxing Huang and Zibin Chen and Weiliang Xu and Zilan Li and Yuanzhi Zhou and Guoyuan Zhou and Wenjing Chen and Xinming Li},
journal= {arXiv preprint arXiv:2510.12528},
year = {2025}
}