English

PokeFusion Attention: A Lightweight Cross-Attention Mechanism for Style-Conditioned Image Generation

Computer Vision and Pattern Recognition 2026-03-30 v3

Abstract

Style-conditioned text-to-image (T2I) generation with diffusion models requires both stable character structure and consistent, fine-grained style expression across diverse prompts. Existing approaches either rely on text-only prompting, which is often insufficient to specify visual style, or introduce reference-based adapters that depend on external images at inference time, increasing system complexity and limiting deployment flexibility. We propose PokeFusion Attention, a lightweight decoder-level cross-attention mechanism that models style as a learned distributional prior rather than instance-level conditioning. The method integrates textual semantics with learned style embeddings directly within the diffusion decoder, enabling effective stylized generation without requiring reference images at inference time. Only the cross-attention layers and a compact style projection module are trained, while the pretrained diffusion backbone remains frozen, resulting in a parameter-efficient and plug-and-play design. Experiments on a stylized character generation benchmark demonstrate that the proposed method improves style fidelity, semantic alignment, and structural consistency compared with representative adapter-based baselines, while maintaining low parameter overhead and simple inference.

Keywords

Cite

@article{arxiv.2602.03220,
  title  = {PokeFusion Attention: A Lightweight Cross-Attention Mechanism for Style-Conditioned Image Generation},
  author = {Jingbang Tang},
  journal= {arXiv preprint arXiv:2602.03220},
  year   = {2026}
}

Comments

12 pages, 5 figures. Revised version with improved method description and corrected references