Model-based sonification based on the impulse pattern formulation

The most common strategy for interactive sonification is parameter mapping sonification, where sensed or defined data is pre-processed and then used to control one or more variables in a signal processing chain. A well-known but rarely used alternative is model-based sonification, where data is fed into a physical or conceptual model that generates or modifies sound. In this paper, we suggest the Impulse Pattern Formulation (IPF) as a model-based sonification strategy. The IPF can model natural systems and interactions, like the sound production of musical instruments, the reverberation in rooms, and human synchronization to a rhythm. Hence, the IPF has the potential to be easy to interpret and intuitive to interact with. Experiment results show that the IPF is able to produce an intuitively interpretable, natural zero, i.e., a coordinate origin. Coordinate origins are necessary to sonify both polarities of a dimension as well as absolute magnitudes.


Introduction
The theory of model-based sonification (MBS) is explained in [1].In parameter mapping sonification, pre-processed data is sonified by using it directly as a control parameter in a signal processing chain.Such a parameter can be the frequency of an oscillator or the inter-onset interval between successive notes [2], or a number of parameters affecting a certain psychoacoustic quantity, such as chroma [3] or fullness [4].In contrast, model-based sonification delivers a framework that defines how user action causes acoustic responses.Typically, such a model remains silent as long as no user interaction happens.
In this paper, we introduce the Impulse Pattern Formulation (IPF) as a model-based sonification approach.Instead of choosing signal processing parameters that shape a sound directly, the IPF has a number of model parameters that deter-Rolf Bader and Robert Mores have contributed equally to this work.

B Simon Linke
Simon.Linke@haw-hamburg.de 1 Hamburg University of Applied Sciences, 22081 Hamburg, Germany 2 Institute of Systematic Musicology, University of Hamburg, 20354 Hamburg, Germany mine the system's behavior.This way, it can mimic natural physical systems.Such a mimicked, natural physical system has the potential to be intuitively understood by users interacting with it.We test this hypothesis in an interactive experiment.We modeled the bow-string interaction of a cello that can produce multiple regimes through the manipulation of a single parameter.Participants control this parameter.Their task is to spot a regime change.Results show that without prior knowledge of the sonification mapping, participants identify the regime change precisely.In sonification applications, such a regime change can serve as a threshold, like a natural zero.
This challenge, of producing a natural zero has been discussed in the sonification literature, [5, chap. 3.6] [6,7].It is necessary to produce both polarities of a data dimension and to sonify absolute magnitudes.In addition, the challenge of intuitive sonification has been treated [8,9], i.e., whether a certain sound characteristic, absolute magnitude, or direction of magnitude change is instantly clear without a learning phase.
social group behavior, and the brain.The IPF has originally been formulated in [10].It describes a system through nonlinearly coupled subsystems.One 'input' subsystem is excited by impulses that reach the other subsystems with delay and attenuation.Here, some energy is reflected, and some is transmitted to other subsystems-again, with delay and attenuation.Eventually, the transmitted impulses either decay completely or feed back into the 'input' subsystem.The IPF can model linear behavior, like resonances, amplitude envelopes and rhythms, and nonlinear phenomena, from enslavement over bifurcation to chaos.
In its simplest form, the IPF is described by Here, g i is the system state at a certain time step i and g i+1 is the succeeding system state.For example, g 0 would be the excitation of the system through an impulse.Choosing an initial value g 0 , Eq. 1 can be calculated iteratively.The time between g i and g i + 1 is not a fixed time interval.It is the time until a new event happens.For example, when modeling a string instrument, it equals the period T 0 of the string's fundamental frequency f 0 , which depends on the sound velocity In other words, after exciting a string through an impulse, the impulse will travel through the string, reflect at both ends of the string and finally return to the input point.When modeling parts of the brain, the time interval could refer to expected response latencies, like p 5 0 [12].The variable α describes the strength of the impulse.The IPF can either diverge or converge to a limit that depends on α.
In a system with more subsystems, the IPF is a recursive equation where n is the number of subsystems, β k is the specific reflection strength of the kth subsystem, g i+1 is the following system state and g i−k are the preceding states.Here, each subsystem could represent a wall, modeling room reverberation.Alternatively, when modeling musical instruments, each subsystem refers to a single or a group of instrument parts, whereby, in contrast to classical generator resonator models (see, e.g., [11]), there is no hierarchical order of the subsystems.An example is described in [13].Here, a dizi flute is modeled.Attachment and detachment of the mirlton membrane is achieved through nothing else but adding one β k that represents the membrane.This makes the sound somewhat sharper and adds a noise floor to the otherwise harmonic sound.Further, e.g., a neural network can be modeled as coupled nuclei, similar to feed-forward and feedback synapses, i.e., afferents and efferents [14].
The IPF models systems in the time domain rather than in the frequency domain.Thus, the IPF inevitably reproduces comprehensive transient behavior.Depending on the control parameters, the IPF may converge to a stable fixed point, produce chaotic time series, or complex periodic oscillations.Whereby not only transitions between those regimes can be observed, but also sudden phase changes.An example is shown in Fig. 1.
Even though the result looks quite similar to the Logistic Map, the IPF does not belong to the same class of equations.As impulses traveling through a musical instrument are usually exponentially damped, the logarithm has to be introduced in the equation.Thus, the unit interval is not mapped onto itself as a singularity occurs, leading to a different overall behavior than the Logistic Map [15,16].However, precisely this nonlinearity of the equation allows the reproduction of complex nonlinear behavior, which is impossible with classical time-based modeling approaches like, e.g., digital waveguides (see, [17]).
The IPF considers dynamic systems and processes as nonlinearly coupled subsystems excited by impulses, just like the finite element method considers systems as mechanically coupled pieces excited by forces.Short time intervals can determine the fundamental frequency of a complex tone.Larger time intervals can represent room reflections.How model parameters are determined is explained in [15].You need to know what characteristics your system is supposed to have.Then, you can choose the parameters that produce this behavior for you.

Applications of the impulse pattern formulation
When utilizing the IPF to model specific instruments, all system parameters α and β k must be chosen correctly to represent the instruments' geometry and further a suitable interpretation of g is essential.A vivid example are multiphonics, as discussed in detail by [18].Single-reed instruments can produce sounds with multiple harmonic spectra.Thus, it is plausible, that those effects are related to the bifurcating regions, shown in Fig. 1.The ratio of succeeding g i represents the audible interval: One technique to produce multiphonics is the use of complex fingerings, resulting in separated regions of open and closed fingerholes.Regions of open fingerholes are then represented by additional reflection points β k .Unfortunately, the exact reflection strength cannot be derived directly from measurements, as they represent impedance changes and phase differences due to different acoustic path lengths.However, assuming that multiphonics can be reliably produced independent of initial values and that slow changes in blowing pressure (and thus α) do not change the audible interval, a suitable parameter set can be found numerically.Linke et al. determine modeling parameters for 236 different multiphonics [18].As shown in Fig. 2 it requires a reflection point far away from the excitation point that is strong enough to disrupt periodic motion to produce multiphonics.
A clear example of how to derive the input strength α is the dynamical IPF model of the minimum bow force for bowed string [19].Increasing the force while bowing a sting leads from scratchy and noisy sounds to a region with prominent upper partials.Further increase of the bow force finally leads For most combinations, β 1 < β 2 is crucial and β 1 lies just above 0 to stable Helmholtz motion.This corresponds to a transition from right to left in the bifurcation scenario shown in Fig. 1.Different analytical equations have been given in the past to describe a quasi-stationary transition into Helmholtz motion (e.g., [20][21][22]).All those equations share the same overall structure and only differ slightly in the choice of specific constants.Thus, a similar equation must exist, which describes the transition from stable states to bifurcations in the orbit diagram of the IPF (Fig. 1) and thus provides a dynamical expression of the minimum bow force, which also implies the likely hysteresis characteristics.A numerically simulated annealing approach (see, e.g., [23]) is chosen to determine an expression of α, reproducing the transition into Helmholtz motion from several self-organized measurements.It can be shown that these dynamic results are more accurate than the quasi-stationary approaches of the past.[19] However, the application of the IPF is not limited to musical instruments.Linke et al. modeled the synchronization of musicians to external rhythms [24].Neglecting the neuronal processes of musicians, the IPF in its simplest form (without any β k ) can describe this problem.Then the system state g refers to the tempo of a musician, and α refers to an external tempo of, e.g., a second musician or a metronome.Further models were developed based on this straightforward approach, which may also cover phase-synchronization.In addition, the IPF does not have to synchronize to the exact beats of the external rhythm and may create polyrhythmic patterns of arbitrary complexity.The results were compared in the light of other publications on tapping to external beats.As shown in Fig. 3, similar to human musicians, even the most simple IPF model adapts to step changes in tempo after a short, chaotic transition and can predict regular tempo changes and quickly adapt to them. the results of this model can be sonified straight forward.Examples are provided by [25].

Towards an IPF-model-based sonification
Being such a straightforward and intuitive physical modeling approach, the IPF can be easily applied for model-based sonification.Depending on the use case, the IPF can be implemented as a pure model-based sonification or-as a borderline case-extended by methods of parameter mapping sonification.This becomes vivid when observing the six components setup, dynamics, excitation, initial state, linkvariables, and listener characteristics of the step-by-step definition of an MBS framework after [1, pp.404-408]: • MBS step 1: Model Setup Similar to [1], one can assume a d-dimensional data set with N records.For each record, an independent IPF must be designed.In the most simple approach, this is done by mapping the single dimensions d to the different β k .Depending on the type of excitation (see below), one dimension can also be mapped to α.
The results can be sufficiently vivid and pleasing.Nevertheless, due to the chaotic nature of the IPF, they depend on the data set and might also be distracting and confusing.Then, it might be a suitable solution to use some parameters to distribute the single records in space.For instance, the data of a self-organizing map (see, e.g., [26,27]) can be fed into an IPF model.The position of the data points on the map also determines their position in the sonification model, while the difference to the neighboring neurons determines the amount of chaos of the IPF model, similar to the u-matrix.Section 2.1 shows that a different interpretation of g is necessary for different modeled systems.Thus, this interpretation can also be affected by single dimensions d of the data, resulting in different types of modulations and underlying waveforms.This can be explored using the Pure Data files provided by [28].While the file "1_Study.pd"relies on phase-shifted signals, the files "2_excitation_beta.pd"and "3_SoundDesign.pd"rely on frequency modulation.Further, all three files have the possibility to change the underlying waveform.The files "1_Study.pd"and "3_SoundDesign.pd"also show another degree of freedom of the IPF.Here, the time between succeeding iteration steps, and thus the fundamental pitch, is controlled by a keyboard.This is a crucial modeling parameter of the IPF and can be easily controlled by a single dimension of the underlying data set.

• MBS step 2: Model Dynamics
The model dynamics of the IPF-model are defined by Equation 2 and can be recursively calculated straight forward, resulting in a series of g i representing the dynamical behavior.

• MBS step 3: Model Excitation
As stated in Sect. 2 α determines the input strength of the system.Thus, if a data point is excited (e.g., by a mouse click), the underlying IPF model can be excited simply by changing α from zero to a value capable of sound production.Here it depends on the model setup whether this value of α is fixed for all records or if it is determined by single parameter dimensions d of the data.Figure 4 shows regions capable of sound production in dependency of α and one additional reflection point β.This leads to a more advanced model excitation: While the data dimensions describe the single β k , the excitation strength α can be controlled by a user.In real life, different objects need different forces to get (acoustically) excited.Similarly, different modeling parameters b k require different excitation strength α for chaotic or stable tone production.This can be explored utilizing the Pure Data file "2_excitation_beta.pd" of [28].Here, different piano keyboard keys refer to different values of β, while the velocity refers to the excitation strength.
The values (and thus the system behavior) equal those of Fig. 4. Further, sound examples for different combinations of α and β are given at [28].Here, one can hear that with rising β higher velocities are necessary to produce sound and even higher velocities to produce a stable sound.Further, it becomes vivid that a higher β extends the settling time of a sound.

• MBS step 4: Initial State
The initial state of the IPF, and thus of the sonification model, is defined by a series of initial states [g 0 , g −1 , g −2 , ..., g −n ] depending on the number n of reflection points β k .Usually, the excitation strength α equals zero for all data points.Thus, they remain silent.

• MBS step 5: Model Link-Variables
The result of calculating the IPF is just a series of g i .In Sect.2.1, several examples were given of how this series can be transformed into sound.Nevertheless, countless other examples can be imagined and might be necessary depending on the modeled system.However, as already mentioned above, it is a borderline case, if, e.g., the pitch or the underlying waveform is changed only for a vivid result and more in a parameter mapping approach or if these different interpretations of g are an inevitable part of the underlying model.

• MBS step 6: Listener Characteristics
The listener characteristics can be designed freely according to the use case.As the IPF delivered rich and complex sounds, usually monaural sounds deliver enough information for suitable MBS.Nevertheless, if spatialization is helpful, it can be done in different ways: The position of the sound in the lis- tener space can be derived directly from the underlying data.This could be useful in the example of a selforganizing map described above.Further, spatialization parameters can be derived from the IPF model, as the different reflection points β k are usually distributed in space.This approach is implemented in the Pure Data file In the gray region, sound production is possible (either chaotic or stable).The initial values were the same for all combinations of α and β: g 0 = 1 and g −1 = 2 123 "3_SoundDesign.pd"provided by [28].Here the sound spreads widely across the stereo panorama if the system is chaotic.If the system becomes stable, the resulting sound becomes narrower.
Taking these steps as guidelines (and maybe relying on the Pure Data files of [28]), the IPF can be implemented for different kinds of sonification purposes.For example, an IPF model was implemented into the "Tiltification App" [29,30], a multi-modal spirit level application for smartphones: Here, the angle of the smartphone controls α.If it rises above 5 • , the IPF becomes chaotic, and the chaos increases as the angle does.A .apk file of this application can also be found at [28].
However, the approaches described above can be easily extended.Think about the IPF to model the synchronization of musicians to external rhythms (see [24] and above).This model can help musicians keep tempo variations small, because the system can adapt to gradual tempo changes, but not to quick ones.Another use case of the same model it to help an athlete perform his exercises at a regular pace without forcing him to a specific tempo.

Method
As stated in Sect. 1 sonifying absolute values, like a coordinate origin or thresholds, is still a challenge [5, p. 34] [6].Nevertheless, this is inevitable, e.g., when distinguishing between different clusters in a data set.If listeners recognize it reliably, the IPFs transition into chaos or stable states may be a promising threshold or boundary.
Therefore, we decided to conduct a psychoacoustic experiment as suggested in the sonification literature [31][32][33].We invited 5 participants (3 male, 2 female, median age 24) to participate in an interactive experiment.They controlled an IPF model of a bowed string (see, [19]).It is based on a simple IPF as described in Eq. 1 with a sawtooth-like impulse applied.The utilized Pure Data file "1_Study.pd"can be found at [28].A keyboard controls the model.The different keys control the pitch, while a ribbon controller (length: 7.4 cm) adjusts α (and thus the bow force) between 0 and 1.63672.
The participants were asked to play arbitrary notes or phrases on the keyboard using their right hand while they controlled the ribbon controller (and thus α) with their left hand.Their task was to start at maximum α (the right end of the ribbon controller) and glide down to spot a "threshold".We did not give any further explanation of the threshold.In the psychoacoustic literature, such a task is referred to as the "Method of Adjustment" [34, chap.1].

Results
Figure 5 shows the trajectories of all participants.They all move down the ribbon controller quite linearly.Once the bifurcation threshold is surpassed, they carry out microcorrections to spot the exact threshold that separates the Helmholtz regime from the bifurcation regime.The participants needed between 5 and 30 seconds to spot the threshold with a precision of 0.02383 ± 0.01878, which corresponds to 1 mm.Two participants (red and orange) instantly switched from macro motion to micro motion.One participant (green) slightly surpassed it twice before switching from macro to micro corrections.One participant (blue) changed the direction of the motion as if he or she was uncertain.But when finally surpassing the threshold, the person also instantly switched from macroscopic to microscopic Fig. 5 Trajectories of the 5 participants.They mildly overshoot the threshold and then carry out micro-corrections to spot the exact point motions.Finally, one participant (purple) crossed the threshold and then returned to the starting point several times, ending up very close to the threshold.The large jumps at around 210 are an artefact.Here, the participant's finger lost contact with the ribbon controller, causing the MIDI value to return to its initial value.However, the participant corrected this quickly.
Without prior knowledge of the sonification principle, and without any indication of which sonic attribute to focus on in order to recognize the threshold, all participants intuitively identified the regime change as the threshold and found it with very high precision and little variance.

Discussion
In the Sonification Handbook, [8] states: "From the side of the sonification itself, the most important question is how to create metaphors that are convincing to the user, need little explanation, are in unison with the user's expectation and create sounds so rich in complexity that users are not bored or annoyed by them."[8, p. 106] As an approach to a solution, the authors suggest "(…) to adopt ideas from physical modelling, or directly (…) use Model-Based Sonification (…) and trust that with learning the user will discover the relevant bindings between data variables and sonic characteristics." We believe that the IPF is such an approach.The experiment results indicate that users intuitively understand the sonification and find the threshold with high precision in a short amount of time.
One strength of the IPF is that it can mimic the behavior of natural physical systems, therefore, sounding rich and natural, and intuitively making sense to a listener.With a similar argumentation, [35] implements a physical model of rectangular plates, from which users could estimate size, material, and alike, through knocking.
Another strength of the IPF is that it can have inherent regimes.Within each regime, it has a certain behavior.Without the explicit formulation of boundaries or thresholds, sudden regime changes may occur.As described above, this is a well-known phenomenon when playing musical instruments.Low bow pressure leads to a noise regime, higher pressure produces bifurcation, and an even higher bow pressure produces the typical Helmholtz regime until the highest bow pressure determines the fundamental frequency by enslaving the string.This behavior also occurs when the force applied to an object is continuously increased.At a certain threshold, the response may switch from pushing to deformation.Blowing up a balloon suddenly switches from growing to bursting.As a glass falls from an increasing height, you hear the eigenfrequencies when the glass hits the ground with increasing amplitude.Until a regime change, when the height exceeds a certain threshold, and the glass breaks.
A third strength of the IPF is that the model can be designed to exhibit the desired behavior first.Then, it can be excited with any desired input sound, changing the audible outcome without affecting the overall model behavior.This way, the IPF decouples the system architecture from the sound design.One can optimize the model behavior first.And then, when the system behaves as expected, different impulse shapes can be explored and optimized concerning aesthetic appeal, naturalness, and other aspects.Nevertheless, one can also apply the IPF as a creative tool for sound design.The underlying dynamic could be entirely made up by messing around with the control parameters α and β k .This trial-and-error heuristic is applied in Frequency Modulation (FM) synthesis, too [36].Finding the right architecture that produces the desired model behavior can have high computational costs.But once the necessary parameters have been identified, even a complicated IPF model can be applied in real-time with much lower computational costs than many physical modeling approaches, such as finite elemente models.
A weakness of the IPF is that the parameters to build the model are not directly linked to physical boundary conditions or computation logic.You must learn to interpret the meaning of the parameters for every single use case, and then choose an appropriate architecture and parameter magnitudes.Therefore, we recommend playing around with the IPF Pure Data files [28] and building your own IPF systems to model how single water drops may turn into rain, how pushing an object turns from translation to deformation, and how blowing a flute turns from noise to overblowing.
Hermann [37] hopes for "other possible yet-to-be-discovebreakred linkages between data and sound".The IPF offers such a linkage between data and sound.Depending on the mode design, the IPF can sound like a musical instrument, but also like any other transient system.More exploration and experimental evaluation are necessary to provide evidence for the suitability of the IPF as a model-based sonification method, to validate the intuitiveness of IPF models during interactive use, and to explore how the same model is perceived with different impulse shapes.

Conclusion
In this paper, we introduced the Impulse Pattern Formulation (IPF) as a model-based sonification method.The IPF is a framework to build a system by means of nonlinearly coupled subsystems that are excited by impulses.The delay and attenuation of coupling, as well as the number and interconnections of subsystems determine its behavior.In addition, the impulse shape has a strong influence on the resulting sound.So far, the IPF has been evaluated as a means to model the behavior of musical instruments and human synchronization to a rhythm.In this paper, we suggest its use for interactive sonification due to its natural, physical behavior, which promises intuitive use.Our experiment provided evidence of an intuitive linkage between data and sound, i.e. that the IPF model can be intuitive and interpretable.Additional experiments are necessary to reveal the full potential of the IPF as a model-based sonification approach.
Funding Open Access funding enabled and organized by Projekt DEAL.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material.If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.To view a copy of this licence, visit http://creativecomm ons.org/licenses/by/4.0/.

Fig. 1
Fig. 1 Bifurcation scenario of the IPF in its most simple form.For each α ∈ [0.36, 1] 2000 iteration steps are performed, and the last 250 values are plotted.Below 1/α = 2 the IPF converges to a stable limit, higher values show bifurcations and chaos, and finally, above 1/α = e the IPF diverges

Fig. 3
Fig. 3 Several scenarios of different tempo changes applied to an IPF which considers phase differences: a step change from 120 to 100 bpm, b linear change from 120 to 130 bpm, c 5% Brownian noise added to a

Fig. 4
Fig.4 Stability of the IPF in dependency of α and a single β.In the gray region, sound production is possible (either chaotic or stable).The initial values were the same for all combinations of α and β: g 0 = 1 and g −1 = 2