Harmonizing Pixels and Melodies: Maestro-Guided Film Score Generation and Composition Style Transfer
Abstract
We introduce a film score generation framework to harmonize visual pixels and music melodies utilizing a latent diffusion model. Our framework processes film clips as input and generates music that aligns with a general theme while offering the capability to tailor outputs to a specific composition style. Our model directly produces music from video, utilizing a streamlined and efficient tuning mechanism on ControlNet. It also integrates a film encoder adept at understanding the film's semantic depth, emotional impact, and aesthetic appeal. Additionally, we introduce a novel, effective yet straightforward evaluation metric to evaluate the originality and recognizability of music within film scores. To fill this gap for film scores, we curate a comprehensive dataset of film videos and legendary original scores, injecting domain-specific knowledge into our data-driven generation model. Our model outperforms existing methodologies in creating film scores, capable of generating music that reflects the guidance of a maestro's style, thereby redefining the benchmark for automated film scores and laying a robust groundwork for future research in this domain. The code and generated samples are available at https://anonymous.4open.science/r/HPM.
Keywords
Cite
@article{arxiv.2411.07539,
title = {Harmonizing Pixels and Melodies: Maestro-Guided Film Score Generation and Composition Style Transfer},
author = {F. Qi and L. Ni and C. Xu},
journal= {arXiv preprint arXiv:2411.07539},
year = {2024}
}