Play as You Like Timbre-enhanced Multi-modal Music Style Transfer 模型架构讲解

"Play as You Like: Timbre-enhanced Multi-modal Music Style Transfer" is a model architecture that aims to transfer the style of music from one track to another while preserving the original timbre (sound quality) of the target track.

The architecture consists of three main components:

1. Timbre Encoder: This component takes the audio waveform of the target track as input and encodes it into a timbre latent space. The encoder consists of a series of convolutional layers followed by fully connected layers. It learns to extract the timbre-related features from the input audio.

2. Style Encoder: This component takes the audio waveform of the source track as input and encodes it into a style latent space. Similar to the timbre encoder, it consists of convolutional and fully connected layers. It learns to capture the style-related features from the input audio.

3. Decoder: This component combines the timbre and style latent spaces to generate the output audio waveform. It takes the encoded timbre and style features as input and decodes them into the output waveform. The decoder also consists of convolutional and fully connected layers.

The training process follows a two-step approach. In the first step, the timbre encoder and decoder are trained using a reconstruction loss, which measures the similarity between the original and reconstructed audio waveforms. This step ensures that the timbre of the target track is preserved during the style transfer process.

In the second step, the style encoder is trained using an adversarial loss. A discriminator is introduced to distinguish between the encoded style features and real style features. The style encoder aims to generate style features that are indistinguishable from the real ones, while the discriminator tries to correctly classify the encoded features. This adversarial training helps the style encoder to capture the style-related features accurately.

During the style transfer process, the timbre encoder is fixed, and only the style encoder and decoder are used. The style features of the source track are encoded using the style encoder and then combined with the timbre features of the target track. The combined features are decoded into the output audio waveform using the decoder.

Overall, the Play as You Like model architecture combines the timbre and style features to achieve music style transfer while preserving the original timbre of the target track