HRPE: Hierarchical Relative Positional Encoding for Transformer-Based Structured Symbolic Music Generation

Abstract

Musicians often structure their compositions hierarchically to imbue their music with rich expressiveness. As a result, generating musically meaningful music with well-organized structures has been a significant research goal for many scholars. Several approaches have been proposed to achieve this objective, typically involving multi-step generation pipelines or sophisticated model architectures based on domain knowledge, which can increase model complexity and generalization difficulty. In this study, we demonstrate that a hierarchical positional encoding adapted for music is sufficient to enhance model performance and generate coherent music with hierarchical structures. We incorporate hierarchical positional information into the Transformer model by modifying the attention matrix with relative position bias at different levels, enabling the model to learn long-short-term dependencies jointly and become less sensitive to positional shifts of several notes. Additionally, we investigate the design of section-level relative positional encoding through ablation studies. To validate our approach, we annotate two datasets (POP909-S and POP2000-S) with music sections and present evidence for both single-track monophonic music and multi-track polyphonic music generation tasks. Experimental results demonstrate that our approach outperforms state-of-the-art Transformer models in both subjective and objective evaluations. We plan to release the source code and annotated datasets upon acceptance.