Material reconstruction from a photograph is a key component of 3D content creation democratization. We propose to formulate this ill-posed problem as a controlled synthesis one, leveraging the recent progress in generative deep networks. We present ControlMat, a method which, given a single photograph with uncontrolled illumination as input, conditions a diffusion model to generate plausible, tileable, high-resolution physically-based digital materials. We carefully analyze the behavior of diffusion models for multi-channel outputs, adapt the sampling process to fuse multi-scale information and introduce rolled diffusion to enable both tileability and patched diffusion for high-resolution outputs. Our generative approach further permits exploration of a variety of materials which could correspond to the input image, mitigating the unknown lighting conditions. We show that our approach outperforms recent inference and latent-space-optimization methods, and carefully validate our diffusion process design choices. Supplemental materials and additional details are available at: https://gvecchio.com/controlmat/.
@article{arxiv.2309.01700,
title = {ControlMat: A Controlled Generative Approach to Material Capture},
author = {Giuseppe Vecchio and Rosalie Martin and Arthur Roullier and Adrien Kaiser and Romain Rouffet and Valentin Deschaintre and Tamy Boubekeur},
journal= {arXiv preprint arXiv:2309.01700},
year = {2024}
}