Audiovisual Cartography: Established and New Multimedia Approaches to Represent Soundscapes

Abstract

Since the mid-1990s, sound has been discussed and used in multimedia cartography. There are four main variants of auditory map elements that have been established in the theory and practice of audiovisual cartography, i.e. abstract sounds/abstract sound sequences, speech, music and, especially, audiorealistic sequences representing so-called “soundscapes”. In cartography, soundscapes are often addressed in large-scale representations. The term originates in multidisciplinary landscape research. Empirical findings of landscape research, and especially those of social constructivist landscape research, have shown the relevance of non-visual stimuli for people’s individual impressions and meanings of the experienced landscape. Amongst the non-visual landscape dimensions, the auditory dimension is the most prominent one. As 3D cartography offers new methods and techniques of designing and experiencing highly realistic, incl. photo- and audiorealistic landscape representation, this discipline becomes more and more interesting for simulating and presenting multisensory landscapes and for presenting the results of empirical findings in landscape research. After an introduction to traditional means of audiovisual cartography and the relevance of auditory stimuli for social constructivist approaches of landscape research, a modern software-based method is presented which highlights the opportunity to imbed 3D sound data representing a location’s soundscape into audiovisual 3D environments in Virtual Reality (VR). It is technically based on the cross-platform game engine Unity3D.

Zusammenfassung

Audiovisuelle Kartographie: Etablierte und neue Methoden zur Repräsentation von Klanglandschaften

Seit Mitte der 1990er Jahren wird die akustische Dimension in der Multimediakartographie diskutiert und eingesetzt. Seither haben sich hauptsächlich vier Ausprägungen durchgesetzt, die als akustische Kartenelemente verwendet werden: abstrakte Töne und Tonfolgen, Sprache, Musik und audiorealistische Sequenzen zur Darstellung so genannter soundscapes (dt. „Klanglandschaften“). In der Kartographie beziehen sich Darstellungen von Klanglandschaften häufig auf großmaßstäbige Raumausschnitte. Der Begriff entstammt der multidisziplinären Landschaftsforschung. Empirische Erkenntnisse der Landschaftsforschung, insbesondere der sozialkonstruktivistischen Landschaftsforschung verdeutlichen, dass nicht-visuelle Reize der Landschaft die individuelle Interpretation der erfahrenen Landschaft maßgeblich beeinflussen. Unter den nicht-visuellen Landschaftsdimensionen gilt die akustische Dimension als bedeutendste. Aktuelle Entwicklungen zu Methoden und Techniken der 3D-Kartographie bieten Möglichkeiten zur hochrealistischen Gestaltung und Erfahrung audio-visueller 3D-Landschaften. Dies kann die Landschaftsforschung durch moderne multisensorische Darstellungsformen sowie Ergebnispräsentationen unterstützen. Dieses Paper führt ein in traditionelle Darstellungsvarianten der audiovisuellen Kartographie und diskutiert die Bedeutung der Klanglandschaften für die (sozialkonstruktivistische) Landschaftsforschung. Im Anschluss wird ein aktuelle Software-gestützte Methode zur Implementierung von audiorealistischem 3D-Sound in 3D-Landschaften in Virtual Reality (VR) vorgestellt. Dies basiert auf der plattformübergreifenden Spiele-Engine Unity3D.

Introduction

The origins of using sound in maps are often associated with John B. Krygier. His pioneering research article entitled “Sound and Geographic Visualization” (Krygier 1994)—inspired by his own seminar paper in Psychology (Krygier 1991)—fundamentally changed a traditional point of view on making maps and cartographic communication. His set of “abstract sound variables”—influenced by Jacques Bertin’s (1967) six graphical variables—gave a ‘multisensory impulse’ to theoretical and practical cartography. The typology of Krygier’s nine abstract sound variables suggests that spatially represented information can be differentiated by specific attributes of sound, i.e. location (the position of a sound), loudness (the magnitude of a sound), pitch (the highness or lowness of a sound), register (the relative location of a pitch in a given range of pitches), timbre (the general prevailing quality or characteristic of a sound), duration (the length of time a sound is or is not heard), rate of change (the varying duration of a sound over time), order (the sequence of sound over time) and attack/decay (the time it takes a sound to reach its maximum/minimum). Perceivable changes/differences of these sound variables likely—a clear empirical verification is still missing by today (Ballatore et al. 2018; Lammert-Siepmann et al. 2017; Schiewe 2015)—facilitate a proper decoding of auditorily encoded spatial information.

Krygier’s idea of cartographically representing ordinal and nominal (geographical) data using sound features, i.e. spatio-temporal auditory map animations, was published at a time where cartography and neighbouring disciplines rapidly shifted from analogue to digital solutions (MacEachren 2004; Müller et al. 2001; Müller and Laurini 1997). The formation of information societies, going along with the computerization of the job market and private households, had caused the fast and competitive development of digital (multi-)media products as well as mass-market software. Animation (flagship) software, such as “Future Splash Animator” or “Macromedia Flash”, allowed the ‘programming’ of maps, instead of only drawing them as usual (Scharlach 2002).

The paradigm shift from analogue to digital cartography required new methodological fundamentals in cartography (see also Brauen 2006; Buziek 1999; Müller 1997). To create an audiovisually animated map, cartographers had the new task to handle (such as record or find and download, cut and edit) sounds and to implement these sound elements into interactive map frameworks—by writing (usually object-oriented) programming/scripting code.

From the mid-1990s onwards, a large bouquet of audiovisual maps has been published in academic papers and launched online—many of these maps were obviously influenced by animated spatial visualizations released by other creative industries, such as animated cartoons (Garfield 2012; Harrower 2004; Peterson 1999) and video/computer games (Edler et al. 2018b; Edler and Dickmann 2016, 2017; Ahlqvist 2011). Nevertheless, compared to purely graphical maps (originating in traditional methods of map-making), the today existing number of audiovisual maps is still very limited (Dickmann 2018, p 170). Without the possibility of considering each of these audiovisual cartographic publications, a set of specific variants of cartographic sound elements has become apparent: (1) computer-generated abstract sounds (and sound sequences), (2) human or computer-simulated speech, (3) recorded or computer-generated music and (4) ‘audiorealistic’ recordings of the real ‘soundscape’ (or computer-simulated imitations).

The most ‘geographical’ of these sound variants in audiovisual cartography is the audiorealistic representation of the “soundscape”, a term coined by Schafer (1977). The term represents the compilation of perceivable sounds in the physical landscape at a specific place over a specific period of time. The possibility of recording the soundscape allows a very close and detailed relation between the map and the multisensory landscape represented in the map (Edler et al. 2012; Laakso and Sarjakoski 2010). The represented soundscape may also document different experiences made by individuals and groups of society (see also Wissmann 2014). Thus, the soundscape and its technical representation based on audiovisual cartography can be closely related to social constructivist approaches of landscape research (Edler et al. 2018a).

This article is dedicated to the linkage between the meaning and relevance of soundscape for landscape research and modern interactive visualization techniques of audiovisual cartography (in 3D). Before going into these details, the established variants of sounds which have traditionally been used in audiovisual maps are introduced.

Examples of Established Sound Variants in Audiovisual Cartography

This chapter intends to give a short and compact overview of the four established variants of sound in audiovisual cartography. Some examples are contained in figures to offer, at least some, additional visual impressions.

Abstract Sounds

The use of abstract sounds (simple tones) in cartography is a plain and pure way of implementing acoustic elements. On the one hand, it can be used to convey semantic information in the map, such as changes in noise emission/noise pollution (Schiewe and Weninger 2012; Kornfeld 2008; Scharlach 2002), positional accuracy of address locations (Bearman and Lovett 2010) or height information in topographic or physical map—the higher the terrain, the higher the tone (Schito 2011; see also Schito and Fabrikant 2018; Sobue et al. 2010). On the other hand, abstract sounds in audiovisual maps are also used to simply signalize that the content has just changed in an animated sequence. This signalizing function of abstract sounds was used by Krygier (1993, published online: 2012), who created a map showing variances in the nationwide distribution of Democratic or Republican college landslides in US elections. When the year changes in the map sequence, an abstract sound is played to indicate this change, going along with the display of a different electoral result. The years documented are additionally graphically represented by a time bar. Hence, the signalizing tone is a redundant feature in the audiovisual map (see Fig. 1).

Fig. 1

Abstract sounds mark the change of the election year in the USA by Krygier (1993)

Speech

Speech, either a recorded human or computer-generated voice, is a multifaceted and adaptable tool in multimedia cartography (see also Dickmann 2018; Lammert-Siepmann et al. 2014, 2017; Morrison and Ramirez 2001). It addresses the linguistic code of transferring facts or concepts. Users of mobile navigation systems know that spoken instructions help people to improve the effectiveness and efficiency of getting from A to B. Apart from ‘acoustically enriched’ satnavs, cartographers have used speech in different scenarios of educational applications: At the beginning, an (original or simulated) voice can give introductory comments of how to use an application, like a ‘guiding agent’ (Edler and Dodt 2010; Borchert 1999; Reagan and Baldwin 2006). These introductory (spoken) words can also help to contextualize the contents of the map application, such as providing background information about history and geography (Pulsifer et al. 2007; Francis 1999).

In the map itself, speech can be linked with specific geographical areas and give information about data and facts that can be read already (redundant information transfer), such as place names, land use categories or precipitation values. Recent cartographic research shows that the additional auditory communication of object names improves not only the memory of names (object-identities), but also the spatial accuracy of their corresponding object-locations (Lammert-Siepmann et al. 2017). The audiovisual communication of semantic attributes of represented spatial objects seems to improve the binding of object identity and object location, which even enhances the spatial accuracy of object-location memory.

Speech can also contain information which is additional to the graphically represented map contents. In the context of learning a foreign language, speech can give, for instance, additional information about pronunciation, grammar and contextualized vocabulary (Edler et al. 2012; Lammert-Siepmann et al. 2011).

Speech can also provide additional information to an entirely different map theme (e.g. for young children), such as (learning) the approximate habitats of prominent worldwide animals and their usual habits and sounds (Edler and Lammert-Siepmann 2012). Figure 2 shows a young child operating with a modern audiovisual globe for children.

Fig. 2

A child is learning animals around the globe based on interaction and audiovisual presentation

Music

Music is a highly complex and diversified sound and rhythm system with its own structural logic (Sonnenschein 2001). Generally, music in maps and map-like applications seems to encourage user pacing, it also transports emotional cues and it can have a motivating function (Brauen 2014; see also Buziek 1999). The sound variant music does not have an equivalent in the visual/graphical dimension of cartography (Edler et al. 2012), unlike abstract sounds (abstract symbols, usually explained in a map key) and speech (map lettering). However, music seems to fascinate some multimedia cartographers and so it is not surprising that musical elements have been used as cartographic elements in some audiovisual maps (Théberge 2005; see also Dickmann 2018, p 171).

Examples exist where music is used in the “intro” of a multimedia cartographic application. In the intro (an opening sequence before the main applications (maps) can be read and used), music usually creates a link between the theme and/or geographical location of the map(s), as known from the film industry, where music is often linked and associated with real or fictional locations (e.g. Star Wars, The Fresh Prince of Bel-Air or the ‘Boom Chica Boom’ song, by Johnny Cash, pointing to the ‘Wild West’ in the opening theme of Bonanza). In other words, music ‘welcomes’ the user/reader/spectator and “invokes a broader geographical context” (Pulsifer et al. 2007, p 208).

A multimedia cartographic example is an intro song sequence of Irish folk in an interactive and audiovisual map tutorial for studying Irish placenames (Edler and Dodt 2010). In an interactive atlas, dedicated to different nautical Antarctica explorations, classical music (String Quartet No. 3 by Wolfgang A. Mozart) introduces the Cook expedition (1772–75). Here, music (of the similar historical period) creates an aesthetic environment (Pulsifer et al. 2007; Siekierska and Armenakis 2007). Accompanying geographical locations and attributes by corresponding music is also a familiar strategy found in video games, where spatial stories are told based on interactive maps or map-like applications, such as Super Mario World or Sim City (Edler and Dickmann 2016, 2017; Grimshaw 2014; Théberge 2005).

In other multimedia cartographic applications, music is also used as a variable encoding and transporting geographical contents which cannot be received from the graphics. In a map on Canada’s international trade relations (1976–2000), musical sequences and their different levels of loudness (see Krygier 1994) represent Canada’s trade balance with different world regions (Brauen and Taylor 2008). Using the ambiguous map title “Rock Music”, different particle sizes determined in sediment analyses of the Matanuska Glacier (Alaska) are represented by song sequences in another audiovisual map (Helmuth and Davis 2004). Compared to other sound variants, music is however rarely used in audiovisual cartography.

Audiorealstic Recordings/Simulations of the ‘Soundscape’

By ‘nature’, audiorealistic recordings or computer-generated simulations of what can be acoustically experienced in the (physical) landscape (“soundscape”, Schafer 1977) are closely linked. The signifier (sound element) and the signified (sound emitting object in the landscape) can have a high level of correspondence (‘iconicity’) in a map (Edler et al. 2012). An audiorealistic sound sequence can ‘echo’ an audible part of the whole concept it is supposed to represent (‘pars pro toto’), comparable to graphic icons (such as pictograms) in mono-sensory (visual-only) maps. An example for this is a recorded sound sequence of a ringing bell (in graphics, for, e.g. a Christian cross) representing locations of Catholic or Protestant churches or monasteries. The other sound variants introduced above would either need a legend to establish a proper cartographic communication (abstract sounds) or could only handle a language-based component to put across the meaning (speech), or would—even in the case of using recordings of a well-known church song—suppose a much higher degree of (socio-)cultural background knowledge to decode and understand the signified object properly.

The example sketched above—ringing bell representing church, but Catholic or Protestant (?)—also indicates the degree of uncertainty carried over by ‘pars pro toto audiorealistic sounds’, which—in a bad case of cartographic communication—may also end up in “unpredictable sound/image interactions” (Caquard et al. 2008). In the review of existing audiovisual maps containing audiorealistic sound elements, it becomes obvious that this variant of sound is more often used in large-scale maps (see also Aiello et al. 2016; Brauen 2014; Thirion 2007; Scharlach 2002)—to guarantee a high level of soundscape appropriateness and to avoid the problem of unpredictable uncertainty (Aiello et al. 2016; Axelsson 2015).

An example is a tourist map for visitors of the Finnish national park “Nuuksio”, north-west from Helsinki. Before visiting the park, people are invited to experience recordings of peculiarities in the landscape cartographically, such as water falling from a dam in spring, or water (still) running in a frozen brook in cold winter times (Laakso and Sarjakoski 2010). Some other audiorealistic recordings are contained in an audiovisual tourist and educational map of the “Landschaftspark Duisburg-Nord” (Edler et al. 2015), a former industrial site in the Ruhr Area, Germany. Today’s leisure activities in the park area, such as diving, climbing and a summer-time cinema, can be heard, by clicking on pictograms. (see more related examples in Brauen 2014 and Scharlach 2002). Figure 3 shows an interactive web-map of Tallinn, Estonia. The markers represent the locations of churches in and around the medieval core of the city. By clicking on the blue markers, the users can get an impression of the soundscapes in and around the church locations. The figure includes a video (audiovisual sequence) and includes a sound of the organ which can be listened to in the dome church of Tallinn (“St. Mary’s Cathedral”).

Fig. 3

Audiorealistic sounds and videos represent the perceivable soundscape in and around the churches of Tallinn’s medieval old town in an interactive map (based on leaflet.js)

The public interest in ‘mapping’ audiorealistic sound sequences in the (physical) landscape becomes visible in such examples of large-scale audiovisual maps. It is also mirrored in social media and real-time cross-platform messengers (e.g. What’s App), video-sharing websites (incl. sound streams, e.g. https://www.youtube.com/) and widely known (free) online sound databases (e.g. https://freesound.org/). Individuals, but also groups with shared interests gather sound events they would like to keep, repeat and exchange. Apparently, audiorealistic sound seems to transport a special meaning to them and is considered as important (see also Papadimitriou et al. 2009).

The meaning and relevance of soundscapes are a research topic which is currently discussed in constructivist approaches of landscape research. Input of this multidisciplinary academic field bring in new discussions to cartography. In cartography, large-scale visualization methods are of rising importance, as these methods help to increase the currently demanded higher levels of realism in the final cartographic media (see Edler et al. 2018c; Hruby et al. 2018; Kersten et al. 2018; Coors et al. 2016; Günther-Diringer 2016; Dickmann and Dunker 2014).

The Meaning and Relevance of Soundscapes

Constructivist approaches of landscape research have established themselves in the last decades (e.g. Aschenbrand et al. 2017; Bruns and Kühne 2015; Gailing 2013; Greider and Garkovich 1994, Daniels and Cosgrove 1988; Cosgrove 1984, 1985; Kühne 2008a, 2009, 2018a). Especially the social constructivist landscape theory is suitable for the formation of ‘multisensory’ impressions. It is originating from phenomenological sociology (Schütz 1960, 1971 [1932]). In comparison to approaches of radical constructivism and discourse theory, where the focus is put on processes of social communication, social constructivist landscape research is rather dedicated to the material world (cf. Kühne 2018b; Weber 2018).

From a constructivist perspective, landscape is not understood as an ‘objectively existing’ object, which it is, in positivism, or as an ‘entity’ from the mutual imprinting of culture and nature, which it is, in essentialism (for more details on the approaches, see Kühne 2018c). From a social constructivist perspective, the constitutive level of landscape is formed by social patterns of interpretation and evaluation. These patterns are incorporated by individuals in the process of socialization. This enables individuals to synthesize the sensory impressions of material objects in their consciousness. It also allows them to communicate via the formed ‘landscape’, without the loss of social recognition (for more details, see Kühne 2018b, d; Wojtkiewicz and Heiland 2012). In the process of synthesizing the sensory impressions of material objects into landscape within the consciousness, not all sensory impressions are considered in the same way. A selection takes place that is based on social conventions, going along with a reduction of complexity (Weber et al. 2018, Burckhardt 2006).

The socialization of social patterns of landscape interpretation and evaluation is a diversified process (for more details, see Kühne 2008a, e; Stotten 2013): The formation of the ‘native regular landscape’ primarily happens during the direct confrontation with phenomena around the parental home, which are then interpreted as ‘landscape’ by formative others (especially parents and relatives). This landscape is formed in a normative way and regarded as ‘normal’. The adolescent is then introduced to more general patterns of interpretation and evaluation of landscape. The mediation is more institutionally bound and influenced by school books, films, photos, paintings, and/or the Internet (see Kühne 2018e; Linke 2017). In this way, a ‘stereotypical landscape’ is shaped out. It represents a ‘common sense’ of an aesthetic, cognitive and functional nature, whereas the ‘native regular landscape’ has a more emotional quality. A third level of landscape socialization is formed by special knowledge of ‘experts’. It is usually formed by academic study. Here, the patterns of interpretation and evaluation are strongly cognitively influenced. They differ a lot from each other, and they are in a distinctive competition with each other in terms of their definitions, but also with respect to the ‘native regular landscape’ as well as the ‘stereotypical landscapes’ (Kühne 2018d).

In addition to its strong emotional linkage, the ‘native regular landscape’ shows clear references to non-visual sensory impressions. It is “filled with first memories of regional language, sounds, smells, colours, gestures, moods and speaking things and deeply anchored in the memory” (Hüppauf 2007 p 112; see also Proshansky et al. 1983; De Visscher and Bouverne-De Bie 2008). Moreover, it offers a “motherly landscaped home and security” (Hard 1969, p 11). Accordingly, stories about early landscape experiences in peoples’ biographies often refer to non-visual sensory stimuli (Kühne and Schönwald 2015; Kühne 2006). In contrast to the significance of non-visual stimuli in the formation of ‘native regular landscapes’, these are largely ignored in most ‘expert special knowledge’. Raab (2001) mentions some ‘quality criteria’ of Western science (freedom of values, general validity and comprehensibility) as justification for this (see also Kazig 2013, 2019; Bischoff 2007; Winkler 2005). The ‘expert special knowledge’ represents an essential basis for the formation of ‘stereotypical landscapes’. In textbooks, but also in documentaries and sometimes also in online videos, they make a significant contribution to the socialization of a positivist understanding of (geographic) space (amongst many: Ipsen 2006; Wardenga 2002; Läpple 1992). Accordingly, stereotypical notions of landscape are more intensively marked by visually perceptible elements than the ‘native regular landscape’. This is likely related to the fact that the human sense of vision is usually much more dominant than the sense of hearing (Tuan 1993). However, in comparison to holders of ‘expert landscape knowledge’, people without a focus on visual aspects rather go along with an acoustic, olfactory, tactile and gustatory dimension (Bischoff 2007; Porteous 1985). Hence, the non-visual dimensions of landscape—even if ‘unseen’—are an important part in the perception and (re-)construction of the entire landscape composite and its evaluation and meaning.

This can also be reconstructed empirically: In 2016, for instance, 447 inhabitants of the Saarland in Germany were asked in a closed questionnaire which aspects of the perceived landscape they considered as relevant for landscape representation. Approximately, two-thirds of the participants stated that sounds were part of the landscape (Kühne 2018d), which is also backed up by findings of similar earlier studies (see Rodaway 2011; Kühne 2006; Vining 1992). The interrelations between visual and non-visual landscape stimuli are particularly addressed in atmospheric landscape research (Kazig 2013, 2019).

In terms of representing non-visual stimuli of landscapes, it is a difficult challenge to grasp and ‘display’ the specific features helping to transport the multidimensionality and multisensory composite of the landscape. Traditional approaches of cartography often implied a ‘visual translation’ of this composite, as landscapes comprising other sensory dimensions were often only graphically represented (Dodt et al. 2017; Lauriault and Lindgaard 2006; Papadimitriou et al. 2009, Müller et al. 2001). Even if non-visual map elements were incorporated, they were usually linked with a 2D or highly generalized 3D visual representation of a landscape (see examples in chapter 2, incl. Figs. 1, 2 and 3). Ongoing developments of digital cartography and ‘virtual world-building’, however, allow a much more (photo-)realistic and flexible (interactive) representation of the visual dimension of landscape (Edler et al. 2018b; Hruby et al. 2018; Kersten et al. 2018): these (photo-)realistic models can be linked with audiorealistic contents. This multisensory mix of spatially referenced stimuli can be merged together in easily accessible audiovisual landscape representations, such as soundscape-featured environments constructed in (immersive) Virtual Reality (VR).

The use of VR appears as an enrichment for social constructivist landscape research and, vice versa, the theoretical framing offered by social constructivist landscape theory can be an enrichment for VR. In the first scenario, empirical settings can be prepared in an idealized form to test the possibilities and limits of theory more targeted. In the second scenario, settings can be constructed theoretically, and empirical outcomes can be integrated into a more general scientific framework of landscape research (Edler et al. 2018c).

In the literature on VR-based landscapes in cartography, and GI sciences, the auditory dimension has not been in the ‘focus’ so far. The following technically oriented chapter shows how 3D-sound can be physically integrated into these ‘walkable’ multisensory 3D-landscapes (approaching a 1:1 scale).

Using 3D Sound in Immersive Virtual Environments

To add 3D sound to a 3D environment which can be accessed in immersive virtual reality, modern software solutions are required that facilitate this linkage between visual and auditory landscape representations. Unity3D is a modern (cross-platform) game engine supporting the implementation of both dimensions by quite user-friendly options. It works with common sound formats such as.mp3, .wav, .aiff and .ogg (for further information on supported formats, see: https://docs.unity3d.com/Manual/AudioFiles.html), which, for instance, can be downloaded in online sound databases (e.g. https://freesound.org/) or generated with customary smartphones or mobile microphone devices.

To enrich a (visual) 3D landscape with 3D sound in Unity3D, two sound-related and invisible components must be added to a scene: an „audio source” and an “audio listener”. The audio source component plays sounds at the position of its so-called “game object”, an object in the visual 3D scene that serves as an anchoring platform for the sound. In the example given in the following, the game object is a car (a virtual replica of an “Opel Manta GT/E”) that ‘carries’ a traditional engine sound to simulate the soundscape of urban traffic (Fig. 4). The audio listener acts like a pair of digital ears at the position of the game object. Both components can be added to any game object in the 3D scene.

Fig. 4

An audio source component is attached to the car. This component is set to play a (repetitive) audiorealistic clip of an engine (car_engine_loop)

The urban soundscape of cars and traffic is an example to illustrate the different patterns of interpretation and evaluation of some parts of society. Depending on the individual (socially influenced) preference (and its intensity) for different means of transport, the evaluation of standing and driving sounds also turn out to be incorrect: A bicycle activist will probably understand the sounds of motorized individual traffic as noise. However, a friend of tuned cars will understand the sound of an Opel Manta (whose exhaust produces an acoustical intensification) as an enrichment of the urban soundscape.

In the audio source component, an audio clip is defined which should be played in the immersive audiovisual VR environment. Additionally, the audio source component contains additional sound options which can be set and changed, such as volume, minimum and maximum distance, and Doppler level (Fig. 4). Within the defined minimum distance, the sound is hearable with the defined volume level. Outside of the minimum distance, the volume will gradually decrease until it reaches zero at the defined maximum distance (behind this ‘virtual 3D border’, the sound is no longer hearable). The Doppler level defines whether and how intensive a Doppler effect is applied. If the value is larger than zero, the played sounds are pitched based on the movement speed and direction of their audio source, relative to the position of an audio listener. It becomes apparent that some dominant variables of Krygier’s typology (1994) also occur in modern (VR-based) 3D animation software.

The audio listener component interprets audio sources around its game object. It calculates a 3D sound mix based on the position and rotation of its game object and the relative position of the surrounding audio sources. The sound mix is then transmitted to the active audio output device (e.g. headset or 5.1 sound system). If an active audio source is placed on the left side of the audio listener, the associated audio clip is played louder and minimally earlier on the left headset speaker than on the right headset speaker. This simulates natural binaural hearing. Usually, the audio source is attached to the main camera in the scene (see Fig. 5). Thus, the position from where the user sees the scene is also the position from where the 3D sound mix is calculated.

Fig. 5

An audio listener component is attached to the camera. Thus, volume and direction of played audio clips are calculated based on the relative position of their game object to the camera

In monitor-based Unity3D applications, the main camera movement and rotation are usually manipulated by mouse and keyboard, or controller input. However, when VR or AR headsets (like HTC Vive or Microsoft Hololens) are used, the main camera position and rotation are usually set to mirror the movements of the headset. Therefore, if the audio listener component is attached to the main camera, volume of played audio clips in each of the used speakers is affected by the head movement of the user (see Fig. 6). This generates a sense of being present in the ‘real physical’ soundscape, where the relative position of objects can be located based on the sounds they are emitting.

Fig. 6

The grey trapeze shows the angle of view of the camera. On the left side of the image, an audio source is directly in front of the camera (which has an attached audio listener). Therefore, the audio clip is played with the same volume in both headset speakers. On the right side of the image, the audio source is to the right of the camera. Therefore, the audio clip is played louder and minimally earlier in the right headset speaker

The same principle of applying 3D sound can be achieved in the game engine Unreal Engine 4. Figure 7 shows 3D sound emitting from a river in a mountain area. In this example, the audiorealistic river sound is distributed sphere-like around a fixed center point of in the river stream.

Fig. 7

Sphere of a 3D sound object in Unreal Engine 4 representing the soundscape in a mountainous river landscape. The orange lines mark the area where the audio source can be heard

Summary and Outlook

This paper has given an overview of the use of different variants of sound established in audiovisual cartography. Amongst abstract sounds and abstract sound sequences (see Krygier 1994), speech and music, audiorealistic sequences representing the “soundscape” (see Schafer 1977, also Krause 2016) of a place are a highly complex entity which can transport and represent a lot of the socially constructed meaning of landscapes.

Current 3D visualization techniques used in modern cartography (cf. Hruby et al. 2018; Kersten et al. 2018; Edler et al. 2018c) go beyond the realism transported via traditional audiovisual maps. It is possible to incorporate audiorealistic sound clips that scatter sound (clips) in 3D and that can move their (dynamic) positions in spatio-temporal animations. These cartographic techniques allow a highly realistic representation of landscapes which can be used interactively and immersively, implying that the user takes an ego-perspective and can navigate through the audiovisual landscape simulation in real time.

Sounds have a special significance for people without expert knowledge in the construction of landscapes. In this respect, from the perspective of social constructivist landscape research, it seems very important to develop modern approaches to map and visualize this dimension of landscape. In terms of the creation of virtual landscapes, a new possibility is offered to explore the significance of the acoustic dimension in a firm way, due to the plannability of the acoustic stimuli by means of VR.

These modern options of cartographic 3D visualization, based on audiorealistic sound sequences and freely accessible game engines, bear a lot of potential for future approaches of landscape research, as these new opportunities of representing ‘multisensory realism’ offer a new dimension of simulated landscape experience and extend the repertoire of media that can be used in (social constructivist) landscape research. In other—and concluding—words, 3D visualization techniques and its immersive use in real time open a link between cartography and new interdisciplinary fields whose researchers had not shown a keen interest in cartography so far, due to the former lack of available innovations to create highly realistic 3D representations of different landscape dimensions.