Remember What You have drawn: Semantic Image Manipulation with Memory
Abstract
Image manipulation with natural language, which aims to manipulate images with the guidance of language descriptions, has been a challenging problem in the fields of computer vision and natural language processing (NLP). Currently, a number of efforts have been made for this task, but their performances are still distant away from generating realistic and text-conformed manipulated images. Therefore, in this paper, we propose a memory-based Image Manipulation Network (MIM-Net), where a set of memories learned from images is introduced to synthesize the texture information with the guidance of the textual description. We propose a two-stage network with an additional reconstruction stage to learn the latent memories efficiently. To avoid the unnecessary background changes, we propose a Target Localization Unit (TLU) to focus on the manipulation of the region mentioned by the text. Moreover, to learn a robust memory, we further propose a novel randomized memory training loss. Experiments on the four popular datasets show the better performance of our method compared to the existing ones.
Cite
@article{arxiv.2107.12579,
title = {Remember What You have drawn: Semantic Image Manipulation with Memory},
author = {Xiangxi Shi and Zhonghua Wu and Guosheng Lin and Jianfei Cai and Shafiq Joty},
journal= {arXiv preprint arXiv:2107.12579},
year = {2021}
}