Vision-language models (VLMs) can describe urban scenes in rich detail, yet consistently fail to produce reliable human preference labels in domain-specific tasks such as safety assessment and aesthetic evaluation. The standard fix, fine-tuning or RLHF, requires large-scale annotations and model retraining. We ask a different question: can a frozen VLM be aligned with human preferences without modifying any weights? Our key insight is that VLMs are strong concept extractors but poor decision calibrators. We propose a three-stage post-hoc pipeline that exploits this asymmetry: (i) interpretable evaluation dimensions are automatically mined from consensus exemplars; (ii) an Observer-Debater-Judge chain extracts robust concept scores from the frozen VLM; and (iii) locally-weighted ridge regression on a hybrid manifold calibrates these scores to human ratings. Applied as UrbanAlign on Place Pulse 2.0, the framework reaches 72.2% accuracy (kappa=0.45) across six perception categories, outperforming all baselines by +11.0 pp and zero-shot VLM by +15.5 pp, with full interpretability and zero weight modification.
@article{arxiv.2602.19442,
title = {UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment},
author = {Yecheng Zhang and Rong Zhao and Zhizhou Sha and Yong Li and Lei Wang and Ce Hou and Wen Ji and Hao Huang and Yunshan Wan and Jian Yu and Junhao Xia and Yuru Zhang and Chunlei Shi},
journal= {arXiv preprint arXiv:2602.19442},
year = {2026}
}