Latent Timbre Synthesis

Abstract

We present the Latent Timbre Synthesis, a new audio synthesis method using deep learning. The synthesis method allows composers and sound designers to interpolate and extrapolate between the timbre of multiple sounds using the latent space of audio frames. We provide the details of two Variational Autoencoder architectures for the Latent Timbre Synthesis and compare their advantages and drawbacks. The implementation includes a fully working application with a graphical user interface, called interpolate_two, which enables practitioners to generate timbres between two audio excerpts of their selection using interpolation and extrapolation in the latent space of audio frames. Our implementation is open source, and we aim to improve the accessibility of this technology by providing a guide for users with any technical background. Our study includes a qualitative analysis where nine composers evaluated the Latent Timbre Synthesis and the interpolate_two application within their practices.

Appendices

Appendix A Constant-Q transform

We can calculate the CQT of an audio recording [31], a discrete time domain signal x(n), using the following formula:

$$\begin{aligned} X^{CQ} (k,n) = \sum _{j=n- \lfloor N_k /2 \rfloor } ^{n+ \lfloor N_k /2 \rfloor } x(j) a_k ^*(j-n+N_k /2) \end{aligned}$$

(2)

where k represents the CQT frequency bins with a range of [1, K], and $X^{CQ} (k,n)$ is the CQT transform. $N_k$ is the window length of a CQT bin, that is inversely proportional to $f_k$ that we define in Eq. 4 Notice that, $\lfloor \cdot \rfloor$ is the rounding towards negative infinity. $a_k ^ *$ is the negative conjugate of the basis function $a_k (n)$ and,

$$\begin{aligned} a_k (n) = \frac{1}{N_k} w \left(\frac{n}{N_k}\right) exp \left[-i 2\pi n \frac{f_k}{f_s} \right] \end{aligned}$$

(3)

where w(t) is the window function, $f_k$ is the center frequency of bin k, and $f_s$ is the sampling rate. CQT requires a fundamental frequency parameter $f_1$, which is the center frequency of the lowest bin. The center frequencies of remaining bins are calculated using,

$$\begin{aligned} f_k = f_1 2 ^ {\frac{k-1}{B}} \end{aligned}$$

(4)

where B is the number of bins per octave.

CQT is a wavelet-based transform because the window size is inversely proportional to the $f_k$ while ensuring the same Q-factor for all bins k. We can calculate the Q-factor using,

$$\begin{aligned} Q = \frac{qf_s}{f_k (2^\frac{1}{B} - 1)} \end{aligned}$$

(5)

where q is scaling factor with the range [0,1] and equals to 1 as the default setting. We direct our readers to the original publication for the specific details of the CQT [31], which also proposed a fast algorithm to compute CQT and inverse CQT (i-CQT), given in Fig. 7.

Fig. 7

A fast algorithm to compute CQT and i-CQT, described in [31] and implemented in Librosa [22]

Appendix B Phase estimation algorithms

Given an audio signal x(n) and its frequency transform X(i),

where N is the total number of GLA iterations, T and IT is the frequency transform and inverse frequency transform function respectively; such as Short-Fourier Transform, or Constant-Q Transform in our case. Note that, the space of audio spectrograms is a subset of the complex number space. The iterative process of Griffin-Lim moves the complex spectrogram of the estimated signal ${\hat{x}}(n)$ towards the complex number space of audio signals in each iteration, as proven in [9].

The Fast Griffin-Lim algorithm (F-GLA) is a revision of the original Griffin-Lim algorithm. A previous study [27] showed that the F-GLA revision significantly improves signal-to-noise ratio (SNR) compared to the GLA, where the setting $\alpha = 1$ (a constant in algorithm 2) resulted in the highest SNR value.

Appendix C Interview questions

• 1. Describe your compositional process when working with the Timbre Space tools

• 2. What was the theme and concept of your composition?

• 3. How did you incorporate the Timbre Space tools into your work?

• 4. How did working with the Timbre Space tools change your composition workflow? What was unique?

• 5. What additional tools/technologies apart from the Timbre Space tools were involved in your work?

• 6. How would you describe the sound qualities of Timbre Space?

• 7. What were the unique aesthetic possibilities of the Timbre Space tools?

• 8. What kind of dataset(s) did you train Timbre Space with?

• 9. If you trained Timbre Space with several datasets, what kind of relationship did you notice between the datasets and the musical results obtained from Timbre Space?

• 10. Did you feel control, and authorship over the musical material generated?

• 11. Did you achieve the aesthetic result you intended?

• 12. What were the positive aspects when working with the tool?

• 13. What were the frustrations when working with the tool? How can it be Improved?

• 14. Would you use it again (if the above were addressed)?

• 15. For whom else or what musical genres/sectors would this tool be particularly useful (if the criticism was addressed)?