English

Multimodal Event Transformer for Image-guided Story Ending Generation

Computer Vision and Pattern Recognition 2023-01-30 v1 Computation and Language

Abstract

Image-guided story ending generation (IgSEG) is to generate a story ending based on given story plots and ending image. Existing methods focus on cross-modal feature fusion but overlook reasoning and mining implicit information from story plots and ending image. To tackle this drawback, we propose a multimodal event transformer, an event-based reasoning framework for IgSEG. Specifically, we construct visual and semantic event graphs from story plots and ending image, and leverage event-based reasoning to reason and mine implicit information in a single modality. Next, we connect visual and semantic event graphs and utilize cross-modal fusion to integrate different-modality features. In addition, we propose a multimodal injector to adaptive pass essential information to decoder. Besides, we present an incoherence detection to enhance the understanding context of a story plot and the robustness of graph modeling for our model. Experimental results show that our method achieves state-of-the-art performance for the image-guided story ending generation.

Keywords

Cite

@article{arxiv.2301.11357,
  title  = {Multimodal Event Transformer for Image-guided Story Ending Generation},
  author = {Yucheng Zhou and Guodong Long},
  journal= {arXiv preprint arXiv:2301.11357},
  year   = {2023}
}

Comments

EACL 2023

R2 v1 2026-06-28T08:22:16.465Z