Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generation

ArXiv: arXiv:2208.05697

Authors

• Ang Lv (Renmin University of China ) lvangupup@gmail.com
• Xu Tan (Microsoft Research Asia ) xuta@microsoft.com
• Tao Qin (Microsoft Research Asia ) taoqin@microsoft.com
• Tie-Yan Liu (Microsoft Research ) tyliu@microsoft.com
• Rui Yan (Renmin University of China ) ruiyan@ruc.edu.cn

Abstract

Lyric-to-melody generation is an important task in songwriting, and is also quite challenging due to its distinctive characteristics: the generated melodies should not only follow good musical patterns, but also align with features in lyrics such as rhythms and structures. These characteristics cannot be well handled by neural generation models that learn lyric-to-melody mapping in an end-to-end way, due to several issues: (1) lack of aligned lyric-melody training data to sufficiently learn lyric-melody feature alignment; (2) lack of controllability in generation to explicitly guarantee the lyric-melody feature alignment. In this paper, we propose Re-creation of Creations (ROC), a new paradigm for lyric-to-melody generation that addresses the above issues through a generation-retrieval pipeline. Specifically, our paradigm has two stages: (1) creation stage, where a huge amount of music pieces are generated by a neural-based melody language model and indexed in a database through several key features (e.g., chords, tonality, rhythm, and structural information including chorus or verse); (2) re-creation stage, where melodies are recreated by retrieving music pieces from the database according to the key features from lyrics and concatenating best music pieces based on music theories and melody language model scores. Our ROC paradigm has several advantages: (1) It only needs unpaired melody data to train melody language model, instead of paired lyric-melody data in previous models. (2) It achieves good lyric-melody feature alignment in lyric-to-melody generation. Experiments on English and Chinese datasets demonstrate that ROC outperforms previous neural based lyric-to-melody generation models on both subjective and objective metrics.

Generated Samples

Chinese Demo

Example 1

Example 2

English Demo

Example 1

Example 2