We propose RelitLRM, a Large Reconstruction Model (LRM) for generating high-quality Gaussian splatting representations of 3D objects under novel illuminations from sparse (4-8) posed images captured under unknown static lighting. Unlike prior inverse rendering methods requiring dense captures and slow optimization, often causing artifacts like incorrect highlights or shadow baking, RelitLRM adopts a feed-forward transformer-based model with a novel combination of a geometry reconstructor and a relightable appearance generator based on diffusion. The model is trained end-to-end on synthetic multi-view renderings of objects under varying known illuminations. This architecture design enables to effectively decompose geometry and appearance, resolve the ambiguity between material and lighting, and capture the multi-modal distribution of shadows and specularity in the relit appearance. We show our sparse-view feed-forward RelitLRM offers competitive relighting results to state-of-the-art dense-view optimization-based baselines while being significantly faster. Our project page is available at: https://relit-lrm.github.io/.
@article{arxiv.2410.06231,
title = {RelitLRM: Generative Relightable Radiance for Large Reconstruction Models},
author = {Tianyuan Zhang and Zhengfei Kuang and Haian Jin and Zexiang Xu and Sai Bi and Hao Tan and He Zhang and Yiwei Hu and Milos Hasan and William T. Freeman and Kai Zhang and Fujun Luan},
journal= {arXiv preprint arXiv:2410.06231},
year = {2024}
}