Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation

Authors

Botao Yu (Nanjing University) btyu@foxmail.com
Peiling Lu (Microsoft Research Asia) peil@microsoft.com
Rui Wang (Microsoft Research Asia) ruiwa@microsoft.com
Wei Hu^ (Nanjing University) whu@nju.edu.cn
Xu Tan^ (Microsoft Research Asia) xuta@microsoft.com
Wei Ye (Peking University) wye@pku.edu.cn
Shikun Zhang (Peking University) zhangsk@pku.edu.cn
Tao Qin (Microsoft Research Asia ) taoqin@microsoft.com
Tie-Yan Liu (Microsoft Research Asia) tyliu@microsoft.com

^ Corresponding author.

Abstract

Symbolic music generation aims to generate music scores automatically. A recent trend is to use Transformer or its variants in music generation, which is, however, suboptimal, because the full attention cannot efficiently model the typically long music sequences (e.g., over 10,000 tokens), and the existing models have shortcomings in generating musical repetition structures. In this paper, we propose Museformer, a Transformer with a novel fine- and coarse-grained attention for music generation. Specifically, with the fine-grained attention, a token of a specific bar directly attends to all the tokens of the bars that are most relevant to music structures (e.g., the previous 1st, 2nd, 4th and 8th bars, selected via similarity statistics); with the coarse-grained attention, a token only attends to the summarization of the other bars rather than each token of them so as to reduce the computational cost. The advantages are two-fold. First, it can capture both music structure-related correlations via the fine-grained attention, and other contextual information via the coarse-grained attention. Second, it is efficient and can model over 3X longer music sequences compared to its full-attention counterpart. Both objective and subjective experimental results demonstrate its ability to generate long music sequences with high quality and better structures.