TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Method


ArXiv: arXiv:2109.09617

Authors

^ Corresponding author.

Abstract

Lyric-to-melody generation is an important task in automatic songwriting. Previous lyric-to-melody generation systems usually adopt end-to-end models that directly generate melodies from lyrics, which suffer from several issues: 1) lack of paired lyric-melody training data; 2) lack of control on generated melodies. In this paper, we develop TeleMelody, a two-stage lyric-to-melody generation system with music template (e.g., tonality, chord progression, rhythm pattern, and cadence) to bridge the gap between lyrics and melodies (i.e., the system consists of a lyric-to-template module and a template-to-melody module). TeleMelody has two advantages. First, it is data efficient. The template-to-melody module is trained in a self-supervised way (i.e., the source template is extracted from the target melody) that does not need any lyric-melody paired data. The lyric-to-template module is made up of some rules and a lyric-to-rhythm model, which is trained with paired lyric-rhythm data that is easier to obtain than paired lyric-melody data. Second, it is controllable. The design of template ensures that the generated melodies can be controlled by adjusting the musical elements in template. Both subjective and objective experimental evaluations demonstrate that TeleMelody generates melodies with higher quality, better controllability, and less requirement on paired lyric-melody data than previous generation systems.

Generated Samples

We utilize ChordPulse to generate muti-track accompaniment for the generated melodies.
For Chinese demos, we also utilize XStudioSinger to synthesize singing voice. (XStudioSinger currently only supports Chinese lyrics)

Chinese Demo

Example 1

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

melody
melody + accompaniment
singing

Example 2

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

melody
melody + accompaniment
singing

Example 3

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

melody
melody + accompaniment
singing

Example 4

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

melody
melody + accompaniment
singing

Example 5

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

melody
melody + accompaniment
singing

English Demo

Example 1

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

melody
melody + accompaniment

Example 2

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

melody
melody + accompaniment

Example 3

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

melody
melody + accompaniment

Example 4

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

melody
melody + accompaniment

Example 5

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

melody
melody + accompaniment

Random Seed

Example 1:

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

melody
melody + accompaniment

Example 2:

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

melody
melody + accompaniment

Longer Input

Example 1:

This browser does not support PDFs. Please download the PDF to view it: Download PDF.

melody
melody + accompaniment