Multimodal Pretraining, Adaptation, and Generation for Recommendation: A Survey

Qijiong LIU^*, Jieming ZHU^*, Yanting YANG, Quanyu DAI, Zhaocheng DU, Xiao-Ming WU, Zhou ZHAO, Rui ZHANG, Zhenhua DONG
^*Equal contribution.
[Paper]

Abstract

Personalized recommendation serves as a ubiquitous channel for users to discover information or items tailored to their interests. However, traditional recommendation models primarily rely on unique IDs and categorical features for user-item matching, potentially overlooking the nuanced essence of raw item contents across multiple modalities such as text, image, audio, and video. This underutilization of multimodal data poses a limitation to recommender systems, especially in multimedia services like news, music, and short-video platforms. The recent advancements in pretrained multimodal models offer new opportunities and challenges in developing content-aware recommender systems. This survey seeks to provide a comprehensive exploration of the latest advancements and future trajectories in multimodal pretraining, adaptation, and generation techniques, as well as their applications to recommender systems. Furthermore, we discuss open challenges and opportunities for future research in this domain. We hope that this survey, along with our tutorial materials, will inspire further research efforts to advance this evolving landscape.

Citation

@article{liu2024multimodal,
  title={Multimodal Pretraining, Adaptation, and Generation for Recommendation: A Survey},
  author={Liu, Qijiong and Zhu, Jieming and Yang, Yanting and Dai, Quanyu and Du, Zhaocheng and Wu, Xiao-Ming and Zhao, Zhou and Zhang, Rui and Dong, Zhenhua},
  journal={arXiv preprint arXiv:2404.00621},
  year={2024}
}