Accurately modeling millimeter-wave (mmWave) propagation is essential for real-time AR and autonomous systems. Differentiable ray tracing offers a physics-grounded solution but still facing deployment challenges due to its over-reliance on exhaustive channel measurements or brittle, hand-tuned scene models for material properties. We present VisRFTwin, a scalable and data-efficient digital-twin framework that integrates vision-derived material priors with differentiable ray tracing. Multi-view images from commodity cameras are processed by a frozen Vision-Language Model to extract dense semantic embeddings, which are translated into initial estimates of permittivity and conductivity for scene surfaces. These priors initialize a Sionna-based differentiable ray tracer, which rapidly calibrates material parameters via gradient descent with only a few dozen sparse channel soundings. Once calibrated, the association between vision features and material parameters is retained, enabling fast transfer to new scenarios without repeated calibration. Evaluations across three real-world scenarios, including office interiors, urban canyons, and dynamic public spaces show that VisRFTwin reduces channel measurement needs by up to 10× while achieving a 59% lower median delay spread error than pure data-driven deep learning methods.
@article{arxiv.2603.13383,
title = {Taming Vision Priors for Data Efficient mmWave Channel Modeling},
author = {Zhenlin An and Longfei Shangguan and John Kaewell and Philip Pietraski and Jelena Senic and Camillo Gentile and Nada Golmie and Kyle Jamieson},
journal= {arXiv preprint arXiv:2603.13383},
year = {2026}
}