Attribute-based regularization of latent spaces for variational auto-encoders

Abstract

Selective manipulation of data attributes using deep generative models is an active area of research. In this paper, we present a novel method to structure the latent space of a variational auto-encoder to encode different continuous-valued attributes explicitly. This is accomplished by using an attribute regularization loss which enforces a monotonic relationship between the attribute values and the latent code of the dimension along which the attribute is to be encoded. Consequently, post training, the model can be used to manipulate the attribute by simply changing the latent code of the corresponding regularized dimension. The results obtained from several quantitative and qualitative experiments show that the proposed method leads to disentangled and interpretable latent spaces which can be used to effectively manipulate a wide range of data attributes spanning image and symbolic music domains.

Appendices

Appendix 1: Computation of musical attributes

The data representation scheme from [40] is chosen where each monophonic measure of music M is a sequence of N symbols $\left\{ m_t \right\} , t \in [0, N)$, where $N=24$. The set of symbols consists of note names (e.g., A#, Eb, B, C), a continuation symbol ‘__’, and a special token for Rest. The computation steps for the musical metrics are as follows:

• (a) Rhythmic Complexity (r): This attribute measures the rhythmic complexity of a given measure. To compute this, a complexity coefficient array $\left\{ f_t \right\} , t \in [0, N)$ is first constructed which assigns weights to different metrical locations based on Toussaint’s metrical complexity measure [50]. Metrical locations which are on the beat are given low weights while locations which are off-beat are given higher weights. The attribute is computed by taking a weighted average of the note onset locations with the complexity coefficient array f. Mathematically,

  $$\begin{aligned} r(M) = \frac{\sum _{t=0}^{N-1} \mathrm {ONSET}(m_t) . f_t}{\sum _{t=0}^{N-1}f_t} , \end{aligned}$$

  (10)

  where $\mathrm {ONSET} \left( \cdot \right)$ detects if there is a note onset at location t, i.e., it is 1 if $m_t$ is a note name symbol and 0 otherwise.

• (b) Pitch Range (p): This is computed as the normalized difference between the maximum and minimum MIDI pitch values:

  $$\begin{aligned} p(M) = \frac{1}{R} \left( \underset{t \in [0, N)}{\mathrm {max}}(\mathrm {MIDI}(m_t)) - \underset{t \in [0, N)}{\mathrm {min}}(\mathrm {MIDI}(m_t)) \right) , \end{aligned}$$

  (11)

  where $\mathrm {MIDI} \left( \cdot \right)$ computes the pitch value in MIDI for the note symbol. The MIDI pitch value for Rest and ‘__’ symbols are set to zero. The normalization factor R is based on the range of the dataset.

• (c) Note Density (d): This measures the count of the number of notes per measure normalized by the total length of the measure sequence:

  $$\begin{aligned} d(M) = \frac{1}{N} \sum _{i=0}^{N-1} \mathrm {ONSET}(m_t), \end{aligned}$$

  (12)

  where $\mathrm {ONSET} \left( \cdot \right)$ has the same meaning as in Eq. (10).

• (d) Contour (c): This measures the degree to which the melody moves up or down and is measured by summing up the difference in pitch values of all the notes in the measure. Mathematically,

  $$\begin{aligned} c(M) = \frac{1}{R} \sum _{t=0}^{N-2} \left[ \mathrm {MIDI}(m_{t+1}) - \mathrm {MIDI}(m_t) \right] , \end{aligned}$$

  (13)

  where $\mathrm {MIDI} \left( \cdot \right)$ and R have same meaning as in Eq. (11).

Appendix 2: Implementation details

Image-based models For the image-based models, a stacked convolutional VAE architecture is used. The encoder consists of a stack of N 2-dimensional convolutional layers followed by a stack of linear layers. The decoder mirrors the encoder and consists of a stack of linear layers followed by a stack of N 2-dimensional transposed convolutional layers. The configuration details are given in Table 1.

Music-based models For the music-based models, the model architecture is based on our previous work on musical score inpainting [40]. A hierarchical recurrent VAE architecture is used. Figure 14 shows the overall schematic of the architecture, and Table 2 provides the configuration details.

Fig. 14

MeasureVAE schematic. Individual components of the encoder and decoder are shown below the main blocks (dotted arrows indicate data flow within the individual components). $\mathbf {z}$ denotes the latent vector and $\hat{\mathbf {x}}$ denotes the reconstructed measure, $b=4$ denotes the number of beats in a measure and $t=6$ denotes the number of symbols/ticks in a beat. Figure taken from [40]

Table 1 Table showing configurations of the VAEs for the image-based datasets

Training details All models for the same dataset are trained for the same number of epochs (models for both image-based datasets and Bach Chorales are trained for 100 epochs, models for the Folk Music dataset are trained for 30 epochs). The optimization is carried out using the ADAM optimizer [27] with a fixed learning rate of $1\mathrm {e}{-4}$, $\beta _1 = 0.9$, $\beta _2 = 0.999$, and $\epsilon =1$e$-8$.

Table 2 Table showing configurations of the MeasureVAE architecture

All the models are implemented using the Python programming language and the Pytorch⁴ library.

Appendix 3: Additional results

Some additional examples from the image-based datasets are shown in Figs. 15 and 16. The musical scores for AR-VAE generated interpolations from Fig. 11 is shown in Fig. 17.

Fig. 15

Manipulating attributes for three different shapes from the 2-d sprites dataset images using AR-VAE. The interpolations for each attribute are generated by changing the latent code of the corresponding regularized dimension for the original shapes shown on the extreme left. Attributes can be manipulated independently

Fig. 16

Manipulating attributes for ten different digits from the Morpho-MNIST dataset images using AR-VAE. The interpolations for each attribute are generated by changing the latent code of the corresponding regularized dimension for the original digits shown on the extreme left. AR-VAE is able to manipulate the different attributes and is able to retain the digit identity in most cases

Fig. 17

Musical score corresponding to the AR-VAE generated interpolations from Fig. 11. While the attribute values of the generated measures are controlled effectively, the musical coherence is often lost (particularly in the case of Bach Chorales)