Adaptive Musical Control of Time-Frequency Representations

  • Doug Van Nort
  • Phillippe Depalle
Part of the Springer Handbooks book series (SHB)


In this chapter we consider control structures and mapping in the process of deciding upon the underlying sonic algorithm for a digital musical instrument. We focus on control of timbral and textural phenomena that arise from the interaction and modulation of stationary spectral components, as well as from stochastic elements of sound. Given this observation and general design criteria, we focus on a family of sound models that parameterize the stationary and stochastic components using a spectral representation that is commonly based on an underlying short-time Fourier transform (STFT ) analysis. Using this as a fundamental approach we build a dynamic model of sound analysis and synthesis, focusing on a design that will simultaneously lead to musically interesting transformations of textural and noise-based sound features while allowing for control structures to be integrated into the sound dynamics. Building upon well-established adaptive algorithms such as the Kalman Filter, we present a recursive-exponential implementation, and exploit a fast algorithm derivation in order to process both additive data and the full underlying phase vocoder. The model is further augmented to allow for nonlinear adaptive control, pointing towards new directions for adaptive musical control of time-frequency models.




autoregressive moving average


discrete Fourier transformation


extended Kalman filter


infinite impulse response


Kalman filter




recursive exponential short-time Fourier transform


spectral modeling synthesis


state-space representation


stochastic state-space phase vocoder


short-term Fourier transform/short-time Fourier transform


  1. 17.1
    D. Van Nort, M. Wanderley, P. Depalle: Mapping control structures for sound synthesis: Functional and topological perspectives, Comput. Music J. 38(3), 6–22 (2014)CrossRefGoogle Scholar
  2. 17.2
    M. Wanderley (ed.): Mapping strategies in real-time computer music, Organised Sound 7(2) (2002)Google Scholar
  3. 17.3
    D. Wessel: Timbre space as a musical control structure, Comput. Music J. 3(2), 45–52 (1979)CrossRefGoogle Scholar
  4. 17.4
    X. Serra, J.O. Smith: Spectral modeling synthesis: A sound analysis/synthesis system based on a deterministic plus stochastic decomposition, Comput. Music J. 14(4), 14–24 (1990)CrossRefGoogle Scholar
  5. 17.5
    T. Kailath: Linear Systems (Prentice-Hall, Englewood Cliffs 1980)zbMATHGoogle Scholar
  6. 17.6
    J.L. Flanagan, D.I.S. Meinhart, R.M. Golden, M.M. Sondhi: Phase vocoder, J. Acoust. Soc. Am. 38(5), 939–940 (1965)CrossRefGoogle Scholar
  7. 17.7
    J.B. Allen, L.R. Rabiner: A unified approach to short-time Fourier analysis and synthesis, Proc. IEEE 65, 1558–1564 (1977)CrossRefGoogle Scholar
  8. 17.8
    J.A. Moorer: The use of the phase vocoder in computer music applications, J. Audio Eng. Soc. 26(1/2), 42–45 (1978)Google Scholar
  9. 17.9
    M. Dolson: The phase vocoder: A tutorial, Comput. Music J. 10(4), 14–27 (1986)CrossRefGoogle Scholar
  10. 17.10
    G.H. Hostetter: Recursive discrete fourier transformation, IEEE Trans. Audio Speech Signal Process. 28(2), 184–190 (1980)CrossRefGoogle Scholar
  11. 17.11
    D. Arfib, F. Keiler, U. Zoelzer, V. Verfaille, J. Bonada: Time-frequency processing. In: DAFX: Digital Audio Effects, 2nd edn., ed. by U. Zoelzer (Wiley, Chichester 2011)Google Scholar
  12. 17.12
    D. Van Nort, P. Depalle: A stochastic state-space phase vocoder for synhesis of roughness. In: Proc. Int. Conf. Digit. Audio Efects (DAFx 06) (2006) pp. 177–180Google Scholar
  13. 17.13
    S. Dubnov, N. Tishby, D. Cohen: Influence of frequency modulating jitter on higher order moments of sound residual with applications to synthesis and classification. In: Proc. Int. Comput. Music Conf. (ICMC 96) (1996) pp. 378–385Google Scholar
  14. 17.14
    H.D. Thornburg, R.J. Leistikow: Analysis and resynthesis of quasi-harmonic sounds: An iterative filterbank approach. In: Proc. Int. Conf. Digit. Audio Efects (DAFx 03) (2003)Google Scholar
  15. 17.15
    Y. Qi, T.P. Minka, R.W. Picard: Bayesian spectrum estimation of unevenly sampled nonstationary data. In: Proc. Int. Conf. Acoust. Speech Sig. Process. (ICASSP 02) (2002)Google Scholar
  16. 17.16
    A.T. Cemgil, S.J. Godsill: Probabilistic phase vocoder and its application to interpolation of missing values in audio signals. In: Proc. 13th Eur. Sig. Process. Conf., Antalya (2005)Google Scholar
  17. 17.17
    P. Depalle, S. Tassart: State space sound synthesis and a state space synthesiser builder. In: Proc. Int. Comput. Music Conf. (ICMC 95) (1995)Google Scholar
  18. 17.18
    X. Rodet, D. Matignon, P. Depalle: State space models for wind-instrument synthesis. In: Proc. Int. Comput. Music Conf. (ICMC 92), Vol. 277–280 (1992)Google Scholar
  19. 17.19
    S. Tomazic: On short-time fourier transform with single-sided exponential window, Signal Process. 55(2), 141–148 (1996)CrossRefGoogle Scholar
  20. 17.20
    S. Tassart: Infinite length windows for short-time fourier transform. In: Proc. Int. Comput. Music Conf. (ICMC 98) (1998)Google Scholar
  21. 17.21
    R.E. Kalman: A new approach to linear filtering and prediction problems, Trans. ASME J. Basic Eng. 82(1), 35–45 (1960)CrossRefGoogle Scholar
  22. 17.22
    G. Welch, G. Bishop: An Introduction to the Kalman Filter, Tech. Rep. TR 95-041 (University of North Carolina, Department of Computer Science, Chapel Hill 1995)Google Scholar
  23. 17.23
    M.S. Grewal, A.P. Andrews: Kalman Filtering: Theory and Practice (Prentice Hall, Englewood Cliffs 1993)zbMATHGoogle Scholar
  24. 17.24
    H. Thornburg: Detection and Modeling of Transient Audio Signals with Prior Information, Ph.D. Thesis (Stanford University, Palo Alto 2005)Google Scholar
  25. 17.25
    M. Morf, G.S. Sidhu, T. Kailath: Some new algorithms for recursive estimation in constant, linear, discrete-time systems, IEEE Trans. Automat. Contr. 19, 315–323 (1974)CrossRefGoogle Scholar
  26. 17.26
    P. Depalle, G. Garcia, X. Rodet: Tracking of partials for additive sound synthesis using hidden markov models. In: Proc. Int. Conf. Acoust. Speech Signal Process. (ICASSP 93), Vol. 1 (1993) pp. 225–228Google Scholar
  27. 17.27
    A. Moghaddamjoo, R.L. Kirlin: Robust adaptive kalman filtering with unknown inputs, IEEE Trans. Acoust. Speech Signal Process. 37(8), 1166–1175 (1989)CrossRefGoogle Scholar
  28. 17.28
    D. Van Nort: Modular and Adaptive Control of Sound Processing, Ph.D. Thesis (McGill University, Montreal 2010)Google Scholar
  29. 17.29
    D. Menzies: Composing instrument control dynamics, Organised Sound 7(3), 255–266 (2002)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2018

Authors and Affiliations

  1. 1.Computational Arts and Theatre & Performance StudiesYork UniversityTorontoCanada
  2. 2.Schulich School of MusicMcGill UniversityMontrealCanada

Personalised recommendations