Multimodal Music Generation with

Explicit Bridges and Retrieval Augmentation

Baisen Wang1,2, Le Zhuo3, Zhaokai Wang4,3, Chenxi Bao5,
Chengjing Wu6, Xuecheng Nie6, Jiao Dai1,2, Jizhong Han1,2, Yue Liao7†, Si Liu8†
1Institute of Information Engineering, Chinese Academy of Sciences
2School of Cyberspace Security, University of Chinese Academy of Sciences
3Shanghai AI Laboratory    4Shanghai Jiao Tong University    5Music Tech Lab，DynamiX
6MT Lab, Meitu Inc.    7The Chinese University of Hong Kong    8Beihang University

†Corresponding authors

arXiv Code

Abstract

Multimodal music generation aims to produce music from diverse input modalities, including text, videos, and images. Existing methods use a common embedding space for multimodal fusion. Despite their effectiveness in other modalities, their application in multimodal music generation faces challenges of data scarcity, weak cross-modal alignment, and limited controllability. This paper addresses these issues by using explicit bridges of text and music for multimodal alignment. We introduce a novel method named Visuals Music Bridge (VMB). Specifically, a Multimodal Music Description Model converts visual inputs into detailed textual descriptions to provide the text bridge; a Dual-track Music Retrieval module that combines broad and targeted retrieval strategies to provide the music bridge and enable user control. Finally, we design an Explicitly Conditioned Music Generation framework to generate music based on the two bridges. We conduct experiments on video-to-music, image-to-music, text-to-music, and controllable music generation tasks, along with experiments on controllability. The results demonstrate that VMB significantly enhances music quality, modality, and customization alignment compared to previous methods. VMB sets a new standard for interpretable and expressive multimodal music generation with applications in various multimedia fields.


pipeline_v3

Demos

We provide uncurated generated samples for video-to-music generation. For 5 videos from different categories, we provide background music generated from 5 methods.

Video Type VMB (ours) CMT Video2Music M2UGen VidMuse
Game
Documentary
Sport
Animation
Movie

BibTeX

@article{vmb,
  title={Multimodal Music Generation with Explicit Bridges and Retrieval Augmentation},
  author={Wang, Baisen and Zhuo, Le and Wang, Zhaokai and Bao, Chenxi and Wu, Chengjing and Nie, Xuecheng and Dai, Jiao and Han, Jizhong and Liao, Yue and Liu, Si},
  journal={arXiv preprint arXiv:2412.09428},
  year={2024}
}

Acknowledgements

Website adapted from the following template.