English

Your Latent Mask is Wrong: Pixel-Equivalent Latent Compositing for Diffusion Models

Computer Vision and Pattern Recognition 2025-12-08 v1 Graphics Machine Learning

Abstract

Latent inpainting in diffusion models still relies almost universally on linearly interpolating VAE latents under a downsampled mask. We propose a key principle for compositing image latents: Pixel-Equivalent Latent Compositing (PELC). An equivalent latent compositor should be the same as compositing in pixel space. This principle enables full-resolution mask control and true soft-edge alpha compositing, even though VAEs compress images 8x spatially. Modern VAEs capture global context beyond patch-aligned local structure, so linear latent blending cannot be pixel-equivalent: it produces large artifacts at mask seams and global degradation and color shifts. We introduce DecFormer, a 7.7M-parameter transformer that predicts per-channel blend weights and an off-manifold residual correction to realize mask-consistent latent fusion. DecFormer is trained so that decoding after fusion matches pixel-space alpha compositing, is plug-compatible with existing diffusion pipelines, requires no backbone finetuning and adds only 0.07% of FLUX.1-Dev's parameters and 3.5% FLOP overhead. On the FLUX.1 family, DecFormer restores global color consistency, soft-mask support, sharp boundaries, and high-fidelity masking, reducing error metrics around edges by up to 53% over standard mask interpolation. Used as an inpainting prior, a lightweight LoRA on FLUX.1-Dev with DecFormer achieves fidelity comparable to FLUX.1-Fill, a fully finetuned inpainting model. While we focus on inpainting, PELC is a general recipe for pixel-equivalent latent editing, as we demonstrate on a complex color-correction task.

Keywords

Cite

@article{arxiv.2512.05198,
  title  = {Your Latent Mask is Wrong: Pixel-Equivalent Latent Compositing for Diffusion Models},
  author = {Rowan Bradbury and Dazhi Zhong},
  journal= {arXiv preprint arXiv:2512.05198},
  year   = {2025}
}

Comments

16 pages, 10 figures

R2 v1 2026-07-01T08:10:16.583Z