Springer Handbook of Systematic Musicology pp 313-328 | Cite as

# Adaptive Musical Control of Time-Frequency Representations

## Abstract

In this chapter we consider control structures and mapping in the process of deciding upon the underlying sonic algorithm for a digital musical instrument. We focus on control of timbral and textural phenomena that arise from the interaction and modulation of stationary spectral components, as well as from stochastic elements of sound. Given this observation and general design criteria, we focus on a family of sound models that parameterize the stationary and stochastic components using a spectral representation that is commonly based on an underlying short-time Fourier transform (STFT ) analysis. Using this as a fundamental approach we build a dynamic model of sound analysis and synthesis, focusing on a design that will simultaneously lead to musically interesting transformations of textural and noise-based sound features while allowing for control structures to be integrated into the sound dynamics. Building upon well-established adaptive algorithms such as the Kalman Filter, we present a recursive-exponential implementation, and exploit a fast algorithm derivation in order to process both additive data and the full underlying phase vocoder. The model is further augmented to allow for nonlinear adaptive control, pointing towards new directions for adaptive musical control of time-frequency models.

- AR
autoregressive

- ARMA
autoregressive moving average

- DFT
discrete Fourier transformation

- EKF
extended Kalman filter

- IIR
infinite impulse response

- KF
Kalman filter

- MSD
mass-spring-damper

- RESTFT
recursive exponential short-time Fourier transform

- SMS
spectral modeling synthesis

- SSR
state-space representation

- SSSPV
stochastic state-space phase vocoder

- STFT
short-term Fourier transform/short-time Fourier transform

## References

- 17.1D. Van Nort, M. Wanderley, P. Depalle: Mapping control structures for sound synthesis: Functional and topological perspectives, Comput. Music J.
**38**(3), 6–22 (2014)CrossRefGoogle Scholar - 17.2M. Wanderley (ed.): Mapping strategies in real-time computer music, Organised Sound
**7**(2) (2002)Google Scholar - 17.3D. Wessel: Timbre space as a musical control structure, Comput. Music J.
**3**(2), 45–52 (1979)CrossRefGoogle Scholar - 17.4X. Serra, J.O. Smith: Spectral modeling synthesis: A sound analysis/synthesis system based on a deterministic plus stochastic decomposition, Comput. Music J.
**14**(4), 14–24 (1990)CrossRefGoogle Scholar - 17.5T. Kailath:
*Linear Systems*(Prentice-Hall, Englewood Cliffs 1980)MATHGoogle Scholar - 17.6J.L. Flanagan, D.I.S. Meinhart, R.M. Golden, M.M. Sondhi: Phase vocoder, J. Acoust. Soc. Am.
**38**(5), 939–940 (1965)CrossRefGoogle Scholar - 17.7J.B. Allen, L.R. Rabiner: A unified approach to short-time Fourier analysis and synthesis, Proc. IEEE
**65**, 1558–1564 (1977)CrossRefGoogle Scholar - 17.8J.A. Moorer: The use of the phase vocoder in computer music applications, J. Audio Eng. Soc.
**26**(1/2), 42–45 (1978)Google Scholar - 17.9M. Dolson: The phase vocoder: A tutorial, Comput. Music J.
**10**(4), 14–27 (1986)CrossRefGoogle Scholar - 17.10G.H. Hostetter: Recursive discrete fourier transformation, IEEE Trans. Audio Speech Signal Process.
**28**(2), 184–190 (1980)CrossRefGoogle Scholar - 17.11D. Arfib, F. Keiler, U. Zoelzer, V. Verfaille, J. Bonada: Time-frequency processing. In:
*DAFX: Digital Audio Effects*, 2nd edn., ed. by U. Zoelzer (Wiley, Chichester 2011)Google Scholar - 17.12D. Van Nort, P. Depalle: A stochastic state-space phase vocoder for synhesis of roughness. In:
*Proc. Int. Conf. Digit. Audio Efects (DAFx 06)*(2006) pp. 177–180Google Scholar - 17.13S. Dubnov, N. Tishby, D. Cohen: Influence of frequency modulating jitter on higher order moments of sound residual with applications to synthesis and classification. In:
*Proc. Int. Comput. Music Conf. (ICMC 96)*(1996) pp. 378–385Google Scholar - 17.14H.D. Thornburg, R.J. Leistikow: Analysis and resynthesis of quasi-harmonic sounds: An iterative filterbank approach. In:
*Proc. Int. Conf. Digit. Audio Efects (DAFx 03)*(2003)Google Scholar - 17.15Y. Qi, T.P. Minka, R.W. Picard: Bayesian spectrum estimation of unevenly sampled nonstationary data. In:
*Proc. Int. Conf. Acoust. Speech Sig. Process. (ICASSP 02)*(2002)Google Scholar - 17.16A.T. Cemgil, S.J. Godsill: Probabilistic phase vocoder and its application to interpolation of missing values in audio signals. In:
*Proc. 13th Eur. Sig. Process. Conf., Antalya*(2005)Google Scholar - 17.17P. Depalle, S. Tassart: State space sound synthesis and a state space synthesiser builder. In:
*Proc. Int. Comput. Music Conf. (ICMC 95)*(1995)Google Scholar - 17.18X. Rodet, D. Matignon, P. Depalle: State space models for wind-instrument synthesis. In:
*Proc. Int. Comput. Music Conf. (ICMC 92)*, Vol. 277–280 (1992)Google Scholar - 17.19S. Tomazic: On short-time fourier transform with single-sided exponential window, Signal Process.
**55**(2), 141–148 (1996)CrossRefGoogle Scholar - 17.20S. Tassart: Infinite length windows for short-time fourier transform. In:
*Proc. Int. Comput. Music Conf. (ICMC 98)*(1998)Google Scholar - 17.21R.E. Kalman: A new approach to linear filtering and prediction problems, Trans. ASME J. Basic Eng.
**82**(1), 35–45 (1960)CrossRefGoogle Scholar - 17.22G. Welch, G. Bishop: An Introduction to the Kalman Filter, Tech. Rep. TR 95-041 (University of North Carolina, Department of Computer Science, Chapel Hill 1995)Google Scholar
- 17.23M.S. Grewal, A.P. Andrews:
*Kalman Filtering: Theory and Practice*(Prentice Hall, Englewood Cliffs 1993)MATHGoogle Scholar - 17.24H. Thornburg:
*Detection and Modeling of Transient Audio Signals with Prior Information*, Ph.D. Thesis (Stanford University, Palo Alto 2005)Google Scholar - 17.25M. Morf, G.S. Sidhu, T. Kailath: Some new algorithms for recursive estimation in constant, linear, discrete-time systems, IEEE Trans. Automat. Contr.
**19**, 315–323 (1974)CrossRefGoogle Scholar - 17.26P. Depalle, G. Garcia, X. Rodet: Tracking of partials for additive sound synthesis using hidden markov models. In:
*Proc. Int. Conf. Acoust. Speech Signal Process. (ICASSP 93)*, Vol. 1 (1993) pp. 225–228Google Scholar - 17.27A. Moghaddamjoo, R.L. Kirlin: Robust adaptive kalman filtering with unknown inputs, IEEE Trans. Acoust. Speech Signal Process.
**37**(8), 1166–1175 (1989)CrossRefGoogle Scholar - 17.28D. Van Nort:
*Modular and Adaptive Control of Sound Processing*, Ph.D. Thesis (McGill University, Montreal 2010)Google Scholar - 17.29D. Menzies: Composing instrument control dynamics, Organised Sound
**7**(3), 255–266 (2002)CrossRefGoogle Scholar