MITT: Musical Instrument Timbre Transfer Based on the Multichannel Attention-Guided Mechanism

Abstract

Research on neural style transfer and domain translation has clearly demonstrated the ability of deep learning algorithms to manipulate images based on their artistic style. The idea of image translation has been applied to the task of music-style transfer and to the timbre transfer of musical instrument recordings; however, the results have not been ideal. Generally, the task of instrument timbre transfer depends on the ability to extract a separated manipulable instrument timbre feature. However, as the distinction between a musical note and its timbre is often not sufficiently clear, generated samples by current timbre transfer models usually contain irrelevant waveforms. Here, we propose a method of timbre transfer, for musical instrument sounds, capable of converting one instrument sound to another while preserving note information (duration, pitch, rhythm, etc.). The multichannel attention-guided mechanism is used to enable timbre transfer between spectrograms, enhancing the ability of the model guidance generator to capture the most distinguishable components (harmonic components) in the process. The proposed model uses a Markov discriminator to optimize the generator, enabling it to accurately learn a spectrogram’s higher-order feature. Experimental results demonstrate that the proposed instrument timbre transfer model effectively captures the harmonic components in the target domain and produces explicit high-frequency details.