Semantic ambient media—an introduction
The medium is the message! And the message was literacy, media democracy and music charts. Mostly one single distinguishable medium such as TV, the Web, the radio, or books transmitted the message. Now in the age of ubiquitous and pervasive computing, where information flows through a plethora of distributed interlinked media—what is the message ambient media will tell us? What does semantic mean in this context? Which experiences will it open to us? What is content in the age of ambient media? Ambient media are embedded throughout the natural environment of the consumer—in his home, in his car, in restaurants, and on his mobile device. Predominant sample services are smart wallpapers in homes, location based services, RFID based entertainment services for children, or intelligent homes. The goal of this article is to define semantic ambient media and discuss the contributions to the Semantic Ambient Media Experience (SAME) workshop, which was held in conjunction with the ACM Multimedia conference in Vancouver in 2008. The results of the workshop can be found on: www.ambientmediaassociation.org.
KeywordsAmbient media Pervasive computing Ubiquitous computation Smart media Semantic Web 2.0 Collaborative environments
The distribution of media throughout the natural environment implies a paradigm change of how to think about content. Until recently, content could be described as a small number of interdependent entities of information. Audio and video information are presented synchronously to form a medium like TV, while 3D graphics and sound effects can be triggered by user interaction in classical gaming. However, in the age of ambient media, the notion of content extends from a entity thinking towards a mesh of loosely related information entities. This new kind of information is generated by a plethora of sensor networks, smart devices and personalized services, driven by intelligent agents as well as the user himself. But unlike in classical linear media, the importance of each entity to the content is highly dependent on the individual consumer and each entity can be part of multiple kinds of content.
Semantic Ambient Media content is basically a context-sensitive remix from all information that is available for a distinct user with a conscious or subconscious desire at a certain place and time. Ambient media production and consumption is embedded in the natural environment of the user, rather than drawing his attention away from it like TV or gaming do. The consumer actively participates and co-designs any contextual media experience, be it personalized music services or generating location based information for special interest groups. Initiatives such as the smart Web considering location based tagging for web-pages are first incarnations of this development. As one example, lets mention geo-caching, where treasure boxes are hidden in the natural environment, but coordinates and pathways to these treasures can be found on websites (see ).
Def. Ambient Media
Ambient media convey knowledge distributed in time and space throughout the natural environment of consumers through a digital overlay morphing with physical daily objects. The principles of ambient media are manifestation, morphing, collaboration, intelligence, and experience. The content for ambient media relates to artistic expression and aesthetics.
Def. Ambient Media Form
The ambient media form is the particular way in which media exist in the natural human environment and how they manifest themselves, the variety of ambient media assets, the arrangements (composition) of ambient media assets and their sub-components, the specific context of the media as genre, and the ambient assets underlying disappearing hardware and software.
Def. Ambient Media Content
Ambient media content aggregates semantically as ambient content objects. It contains actual knowledge contained in ambient spaces composed of ambient assets in a particular work. Ambient media content is distinct in form and style from traditional media through its principles.
The traditional unidirectional media such as newspapers, television and radio have been challenged by so called social media services, in which audience participation and interactions between the consumers significantly enhance the overall media experience. To date, participation means effort, which is undertaken only by a minority of the overall users, making it the prime scarce resource in the world of social media. In order to foster the emergence of new ambient media services, the convergence between consumer and producer must evolve further to blend consuming, and producing itself. Users shall create content “en passant” while using a service, creating a whole new service experience. Two user requests might target two different sides of a story being the answer to each other—this is where semantics enters the game: each user interaction produces data, but only sophisticated semantic processing can turn it into valuable information.
Evolving miniaturization of communication technology allows for building complex sensor networks, adding even more data to be turned into valuable context information. Tough challenging in terms of technology, the plethora of information is as well the foundation of a kind of media, suited for a restless nomad generation: ambient media with added semantics. Media arts already pioneered to explore its vast potentials and we are sure a commercial avantgarde is soon to follow.
1.1 Principles of ambient media
Related works to define ambient media seen from different perspectives
Media Content and Form: Media & Arts Perspective
What is ‘content’ and how can it be presented in the age of ‘ubiquitous’ and ‘pervasive’?
How to present, select, compose and generate ambient content?
How to manage and re-use ambient content in specific application scenarios (e.g. e-learning)?
What are ambient media in terms of story-telling, interactivity, and art?
Collaboration, Interactivity, Participatory, and Experience: Consumer Perspective
What is interactivity between the single consumers and consumer groups in the ambient context?
How can collaborative or audience participatory content be supported?
Which methods for experience design, prototyping, and business models exist?
Technology Perspective: Enablers
How can existing media such as TV, home entertainment, cinema be extended by ambient media?
What are the characteristics of ambient media, its content, and its technology?
How can sensor data be interpreted and intelligently mined?
Manifestation: ambient media has an actual physical manifestation within the natural environment. Or to put it in other words, ambient media objects are rendered in time and space by stimulating human senses;
Morphing: there is an overlap and binding between the artificially created ambient media objects and real world objects—thus the digital created world overlay and the real world. User interaction triggers to morph content between both worlds creating narrative flows for the consumer. The consumer is immersed into a mix of artificially created and real-world medium;
Intelligence: ambient media content is provisioned through a smart media environment which aggregates content by extensive use of semantic descriptions and sophisticated context reasoning algorithms. Rather than content is consumed via pull/push schemas—adding semantics and system intelligence allows automated content aggregation;
Experience: creating a certain user experience is the art of linear media, where content is composed by creative humans for others. Yet, adding semantic to ambient media does not know static, monolithic content, but only a vast mesh of information entities that may or may not be related to each other in a certain context. Therefore a deep understanding of how each entity stimulates our senses and intellect is vital in order to use this plethora of content for creating myriads of context-tailored experiences. Seeing this point from the information processing point of view, we nowadays process information and data. But we are not aware how we are able to process knowledge and wisdom (e.g. which we give as parents to our children in form of wisdom—“do not touch the oven, as it could be hot and you burn your fingers”). When talking about experience, we are talking about knowledge transferred between parties. The goal of ambient media is the creation of meaningful packages of knowledge ‘nuggets’ and transferring it to the consumer. These knowledge ‘nuggets’ are based on the experience of the consumer—therefore ambient media lead us naturally towards experience orientation, rather than to simple data and information processing media we know nowadays;
Collaboration: user-generated content is a matter of collaboration—however, collaboration not essentially understood in today’s world of Web 2.0 and its services. In the context of ambient media, collaboration addresses not only consumer created content, but also inter-device collaboration, computer mediated collaboration, and inter-human collaboration. Consumers implicitly provide data about their preferences, behavioral patterns, and goals to the machine to achieve goals. The machine—or the distributed net of devices assist the human to obtain content, and act in his behalf.
1.2 Related work
A good starting point for the exploration of ambient media, their possibilities and basic definitions can be found in [19, 20, 22, 34]. The major input for this article where the contributions submitted to the SAME 2008 workshop. The related work is clustered in different perspectives, and depicted in Table 1. It is also worth mentioning that the idea of ambient intelligence—being the ambient media underlying key technology has been defined by the European Commission as research goal for 2010 [15, 16].
2 Why ambient media?
Nowadays ambient media manifest in many, many opportunities. The time is right for the rise of Ambient Media. Several ground-breaking technologies have been developed during the past years. Audiovisual and social web services are established and the digital avantgarde might be ready to look out for the next big thing. Within the scope of this section we are discussing success factors and threat issues for the rise of ambient media. We attempt to define the general framework for the development of semantic ambient media. As ambient media are currently a general concept rather than well defined clearly defined object, we only can pinpoint into a few directions to identify potentials. These are illustrated in Fig. 2 and discussed within the scope of this section.
2.1 Threats issues for ambient media
Information overflow: filtering the noise from data and getting access to the knowledge in the information jungle is one major threat for ambient media. One simple example is spam—data without any information—how can we provide the consumer with a data environment without noise and challenge the information overflow;
Digital divide: age, language, location and economical divides—how to ensure the ambient media is usable by majority of people everywhere?
IPRs and copyrights: is the copyright and IPR system flexible enough and provides the possibilities to launch the system?
Physical information overflow: are the available resources in terms of storage space, network capacity, natural resources, and energy sufficient to deploy ambient media systems?
Social disconnectivity: digitalization leads to a social disconnectivity, where people’s communication is via digital environments and mediators, rather than in real-life settings—this threat of ambient media can be put under the term ‘social disconnectivity’, and will be further on discussed within the scope of this article at a later stage;
Fragmentation and competition: will competing offers of similar devices, services, digital spaces, and communication possibilities be an ‘antagonist’ or ‘protagonist’ of ambient systems?
Acceptance: will consumers and professionals accept the new way of creating and consuming content?
Additional costs: who will pay for all the content for the new media if there is extra need for content because of the additional ambient channels?
Energy consumption: the most obvious threat issue is the energy consumption of data centers and miniaturized devices and sensor networks. The challenge of minimizing energy consumption and carbon footprint of data centers, miniaturized devices, and sensor networks (see e.g. );
Scalability: The constantly increasing amounts of information raises regularly new scalability issues of their processing;
2.2 Success issues for ambient media
Information filtering and analysis: A lot of tools have already been developed based on Semantic Web and data mining technologies that allow an efficient filtering of information and analysis of e.g. sensor data streams;
Personalization in information provisioning: The successful inclusion of individual information needs have already been implemented in a large number projects e.g. APIS—Advanced Personalization in Information Services (see );
Urban landscape as space for content: the urban landscape as environment for embedding content—creating awareness of the connection between synthetic overlay and real physical world by gaining an additional place for embedding content, as e.g. done in the case of “ambient advertising”;
Open information management: information management on an opened basis, with access to information and knowledge for everyone by integrating the collaborative efforts of the Web 2.0 community;
Common technical infrastructures: common standards, service & content discovery mechanisms, user understanding, and service innovation are a success issue for ambient media, and allow the emergence of new base technology platforms;
Critical mass of channel technology: a critical mass of channel technology allows keeping information channels from perishing over time. The main opportunity is to identify the meaning of technologies versus message—and create real-world trials with consumers;
Critical mass of consumers: the most obvious success issue is a critical mass of consumers for any new ambient media service. The number of newly launched media services is growing constantly and most of them loose the fight for the consumer’s attention, even those based on highly promising technology. An anthropology for the digital age will be challenged to analyse the criteria of success and failure of new media.
Understanding contemporary media culture: understanding of the contemporary media culture and its evolution is an essential success factor for ambient services. Thus gaining sensitivity to developments in digital art and its facets might lead to new ‘ambient media cultures’. Technology developers should be well aware of existing ambient art works, as they define the future potential use of an ambient technology. And anthropology—especially “digital anthropology” is obviously needed to find explanation why certain media technologies survive;
Development of tools and methods: how to enable the communities and individuals to contribute in dynamic way? Lower the technology boundary as much as possible, so that non-programmers can start creative work. Making tools open source whenever possible. As an example, the AR Toolkit started as garage development culture in augmented reality ;
A few of the other success issues, such as the connectivity of physical world and synthetic world overlay are discussed at a later stage within the scope of this paper.
3 The role of semantics in intelligent media
In ambient media the presentation and interaction with content goes much beyond the possibilities of the classic media like print, television or the standard World Wide Web. The perception of content has changed from single entity thinking to a plethora of information sources like sensor networks, data streams, multimedia content or services of any kind. The type of content is as manifold as the number of content sources. This generates a growing demand on presenting multimedia content efficiently, adapted to the personal needs and seamlessly integrated into the environment. The underlying principles of such an ambient multimedia were presented previously, and are manifestation, morphing, intelligence, and experience. However, it is not only the integration into the environment but also the way the information, or better the message behind the information, is transferred to the consumer. Therefore, the realization of ambient media requires a good understanding of the semantics of underlying content, e.g. that a series of values represent temperature recordings or if an audio recording is music or a reading. The better the understanding of the underlying semantics is, the better can be the presentation of the content afterwards. This is independent from manual or automatic generation of presentations. For humans it is much easier to understand the semantic of information but they are limited in many ways. Therefore we focus in the following discussion on the machine processing of content and data.
3.1 Representing semantics
Since Tim Berners-Lee et al.  formulated the vision of the Semantic Web the meaning of data plays an increasingly important role in computer science. In the meantime a lot of technologies have been proposed to extract semantics from documents and make them available in a machine processable way. One important issue is to have a model of the world (or normally a part of it) within the machine. Typically ontologies are used for this purpose. In  ontology is defined as an “explicit specification of a conceptualization”, which is “the objects, concepts, and other entities that are presumed to exist in some area of interest and the relationships that hold among them.” Essential is that an ontology specifies concepts and relationships that are necessary to model a domain. Furthermore it provides the representational vocabulary as a link to the real world. For example we are now able to describe that “Satisfaction” is a song of type “Rock” and played by the “Musicians” with the name “Rolling Stones”. To standardize these descriptions also on a syntactic level the Web Ontology Language (OWL)  has been developed by the W3C based on RDF . RDF is standard format for the exchange of resource description on the Web.
With ontologies we have a powerful technology to represent knowledge in a machine readable form. However, the real world is not machine readable and therefore knowledge need to be extracted from documents, audio and video recordings or data streams. The extraction of information from content [8, 9] and the detection of senses  are major research fields and the interested reader is referred to the related literature for details. However, it should be noted that often these annotation are done manually, e.g. by tagging the song “Satisfaction” with the type “Rock”, as extraction technologies for audiovisual data still has a lot of limitations. This kind of “human computing” gained a lot of attention with the Web 2.0 or Social Web. Users are interested to share their favorite resources (songs, ideas, opinions, etc.) with their friends and describe them. This information can be extracted and used to annotate resources .
3.2 Using semantics for preparing ambient media experiences
By applying the previously mentioned technologies to content, the results are content objects annotated with machine readable annotations / metadata describing the content semantics. So the next step towards an ambient media experience is the selection of the content for presentation. Selection goals are either predefined, e.g. present the latest news or user driven, e.g. present the latest album of the Rolling Stones.
As previously mentioned we are not looking anymore on a single data source. So if necessary several resources need to be queried and the results have to be combined. Search among distributed data sources, which are typically also heterogeneous regarding their query language and data structure, is ongoing research field. A survey about automatic schema matching can be found in . The usage of semantics for the data integration is for example used by Bergamaschi et al. in .
Even if a huge set of solutions for all kinds of information and all types of queries already exists, the selection of the “right” information according to the user needs is still under research especially with rapidly increasing amount of information available. However, semantics play different roles in this phase. The first role is to improve the processing of a query. To understand the semantics of query allows expanding it to narrow or broaden the results. So if a user would search for the song “Satisfaction” with a genre type “Heavy Metal” the result might be empty. If the system knows that “Heavy Metal” is a subset of Rock Music, the query can automatically be relaxed to search in the broader genre “Rock”.
3.3 Semantics of user context
In terms of ambient media experience a different semantic could become interesting: the semantics of the user context. The information needs to depend on their current activity. For example if someone poses a query about “Satisfaction” at his office, the implicit information need is to get some background information about the song. If the same person is hearing music while sitting in a bus the implicit need might be to hear the song.
A step further is the analysis of the user behavior. The user is interacting with the system in various ways for example by touching a screen or by moving between tables. The user activities are captured by the system, which afterwards start interpreting the user activities on order to understand the user wishes. Again the machine needs to have a model of possible user activities and related machine activities. Hence such a model allows interpreting the semantic behind the user activity and making it usable for the machine. This enables the machine to react on the user activities (e.g.[2, 41] ). Content related interaction typically result either in a new presentation or view of the same content or in new set of content object that needs to be presented.
The analysis of user activities furthermore allows a personalization of the content selection as well as the content presentation. By observing the user activities and the detection of behavioral patterns the user preferences can implicitly be derived. The results of these analyses can be fed back into the content selection and presentation process. So the user context and activities influence the selection of the content in many ways: (1) the content source (e.g. document library, audio archive), (2) the content selection and (3) the result processing. In the result processing the usage of semantics also enables a more easy composition of different types of content, e.g. for complementing a music recording with additional description about the musicians, criticisms or videos. Such a completion can be based on the information that a song is played by a musician and that songs can have criticisms.
3.4 Using semantics for content presentation
Finally, also the selection of the “best” presentation form depends on the semantics of the content. For example the room lighting during playing a music recording should depend on the music style. This can either be done by an online analysis of the audio data or by using the explicit semantics taken from the content annotations to select matching set of colors. The problem becomes even more complicated if the presented content is a composition of individual content objects, e.g. a music recording with a textual description and a video of the musicians. Furthermore it should not be forgotten that the presentation is not always done in a web browser or standard screens. More sophisticated ways like Roomware  or the Hello.Wall  already exist and require a very specific processing of the information in order to meet their unique presentation and interactions schemes.
Overall the usage of semantic information enables a more flexible usage of ambient media technologies on top of existing media sources of all kinds. Although a lot of research on semantics is still necessary, several tools already exist to work with semantic information that can easily be integrated into new systems to explore the possibilities. The drawback is the investment in the definition of semantics. But the gained flexibility will lead to more intense ambient media experience.
4 Research problems
The form of ambient media is currently not defined, as well as it is poorly defined what actually ambient content is. However, ambient media have to be smart, and only semantic models can help us to understand and assemble the underlying content according to the respective context. The ambient presentation of this content will evoke a certain experience in the user that is the response to a conscious or unconscious desire. Not enough, this experience itself can be conscious or unconscious to the user as well, or even a combination of both. This calls for a new and highly dynamic form of semantics, resulting in one key challenge of ambient media:
Which experience does the perception of a content entity evoke in relation to the individual user, the environment and in combination with other content entities?
What is ‘content’ and how can it be generated, selected, composed and rendered in order to be presented to the user in an ubiquitous media environment?
What is ‘context’, and how can it be described and how does it shape user desires and experiences?
What is the impact on the consumer in terms of attention and obstrusion, both factors which might limit interactive and participative application scenarios and ultimately key factors for a successful experience design?
What are the key technologies driving future sophisticated ambient media services? What are the design principles, methods, algorithms, and set of tools that need to be created?
Which kind of interactivity and narrative paradigms will arise and how will services make use of both conscious and unconscious attention of the user? How will this change and extend existing media such as TV, gaming, cinema, and radio?
Finally, where has media art already pioneered what we are trying to achieve on a commercial scale nowadays?
During the SAME 2008 workshop these questions have been discussed heavily. The workshop participants had backgrounds in technology, art, content creation, media, and human-computer interaction enabling us to provide multidisciplinary answers. A large part of this article is based on the participants’ contributions in form of presentations, group works, and as intense open-space discussions. The results are presented within the scope of this article.
5 Enablers: technology perspective
End to End (E2E) environments
novel media rendering technology
5.1 End to End (E2E) environments
Besides cross-service orientation, another question is how to connect complete environments for integrated communication services? Telepresence system enable the communication between remote locations. However, they rather poorly integrate the ‘feeling’ of both remote environments and communication services and integrating the full environments.
The paper contributed by V. Singh, H. Pirsiavash, I. Rishabh, and R. Jain  points out that the current communication environments are oblivious to and do not really allow people to act naturally. As the key idea, this paper proposes an environment to environment (E2E) communication approach as opposed to a traditional information transfer as is e.g. taking place with common messenger software. Instead of a fixed environment setup (like HP:HALO), E2E allows the use of natural environments by inter-mapping their heterogeneous physical components. The communication system architecture is based on events, which are detected by sensor fusion and time-line segmentation approach. Each environment is abstracted as a peer, and each sensor is seen as a web-service. So called ‘Joint Situation Models’ allow collaboration across environments (in the ‘Joint Space’).
To create an E2E environment, the following design principles apply : natural interaction, semantic interaction, seamless interaction, bi-directional environment connectivity, heterogeneousness, and privacy rights. These design principles have been applied by the team in one telemedicine and one office collaboration setting. In both experiments, environmental models have been created manually. An automated approach still requires further work. However, several E2E principles have been realized in practice with natural interactions and dynamic environment reconfigurations (e.g. when people move from a room to another).
Move towards rich multi-modal interaction in the real-world varying communication environments requires methods and approaches presented. However the challenge is still to enable automated configuration and carry out system-level optimizations to be able to experiment with new application types.
5.2 Mobile technology
One of the core characteristics of ambient media is pervasiveness with use of multiple media channels . In single media format the message is basically received only when the user is within the reach of the media channel or technology. In convergent media world the single channel is disappearing in many cases . It is not evident whether convergence is media or technology driven, but in any case the pervasiveness of ambient media ensures that the message will arrive independent of availability of any single technology.
Mobility and pervasiveness combine elements from technology and message driven ambient media. Mobility is a natural part of our society, but mobile media use shows a heavy technological bias. Different gadgets and computing devices—such as MP3 devices and mobile phones—are the prime focus of our view to mobile media. Yet the most popular mobile services are not driven technologically, but from social needs: for communication, messaging, entertainment, photography, gaming, and provision of personalized services to find its way in our daily data jungle. However, mobile gaming is a main topic in today’s research world, but its consumer adaptation will still require some time. But technology such as GPS, which is currently used for navigation and route sharing will sooner or later also find its way into the gaming world.
As mobile media is socially driven, user generated content is a very relevant part of the mobile ambient media. This increases the pervasiveness, as users can generate content for very special topics and places, eliminating the blind spots of the classical media landscape. One can claim that mobile media is exactly what long tail model describes . The media originates from many small sources—individual users—and end to as many individual destinations. In ambient and pervasive media terms, the challenge is how everything can be connected in technical and semantic terms. Individual users see the relevance in their immediate communication environment, but the big picture is easily lost. A classic example of this is citizen journalism, where professional editors face the huge challenge to filter out the relevant messages from the vast user generated input, although the whole phenomenon is viewed as a major opportunity and disruption in journalism . A major challenge in mobile and pervasive media is how both the content generated by the users as well as the content they consume can be connected to extract information on the user preferences. The solution should not only be feasible in technological sense, but it should make the right content available in any context in delightful and understandable manner. Thus, the challenge is both in technology (data mining, context sensing) and usability (social filtering, user context).
5.3 Novel audio-visual media rendering technology
5.3.1 FogScreen technology for visual rendering
Any media environment needs to render its content and enable a human perceptible form. Many times ambient media are said to be hidden into the natural surroundings of the consumer, however, to be ‘perceivable’ new ways of rendering are required. One example for novel rendering technology are FogScreens, which are described in the paper of I. Rakkolainen [28, 30] as new way for immersing the consumer into an entertainment environment. In the era of ambient media, new display technology is required to present content in a non-intrusive way everywhere. Mid-Air displays are walkthrough displays, whose visual content appears to be floating in the air. Mid-Air displays therefore create a novel possibility how content can be presented, and how it is possible to interact with content, which is e.g. embedded as walkthrough pin-wall on airports [29, 31]. Rather than being purely a visual display, FogScreen technology let’s the consumer interact and allows a certain kind of touch-user-interface with many possibilities for other input devices (e.g. human gestures). It is also suitable to be put into public spaces for displaying art.
5.3.2 Transformation of visual impulses to audio impulses
Ambient environments are multi-modal, thus also the required rendering technology is not solely visual. A more immersive environment in terms of audio is presented by P. Codognet and G. Nouno , describing an interactive sound installation to “turning an ambient dynamic landscape into a real-time generated soundscape”. They turn the light-spectacle of Tokyo’s skyline into a unique ambient music experience. In terms of semantic ambient media, the installation transforms visual impressions into music—thus the installation aggregates the content based on smart imaging algorithms and creates ambient content. It is a nice example how ambient technology can be utilized to generate, transform, and render content in the ambient age.
6 Ambient space—the natural environment as place for media
cross-service oriented spaces
amplified reality to overcome introspection
connectivity between physical world and it is synthetic overlay
automated aggregation of content
public/private spaces for media environments
6.1 Cross-service oriented spaces
Currently we can see trends towards cross-service integration and to provide service spaces, rather than single services or hardware platforms. Examples for this trend are application integrating various platforms, such as social networking sites (e.g. Facebook), messenger applications (e.g. Skype), and Internet radio (e.g. last.fm). Fring  is an excellent mobile application, integrating several applications mentioned above—among others. In the future applications such as Google’s Longitude will enable richer applications integrating more services and the data interchange between them. One step towards this direction is the Portable Personality (P2) project [24, 33, 38], which aggregates user-context data from different application to obtain a personality profile. The P2 application taps consumer data from various clients (e.g. MP3 player, video player) and forwards it to a mobile device. While moving between different environments (e.g. home, office), the profile can be exchanged autonomously between applications (e.g. home HiFi system and office HiFi system) to personalize the digital environment.
6.2 Connectivity between physical world and it is synthetic overlay
When talking about ambient media, we are immersed into the media throughout our physical environment and unconscious traversal through content predominates. Thus, we are either physically or virtually isolated. In ambient media both, the physical and the virtual world get connected—we can speak of a physical world overlaid by a synthetic one (connection of synthetic world and physical world). An excellent attempt to overcome the described issues are artistic installations embedded into public spaces and urban environments or performances enabling collaboration during dance performances (see  and ).
6.3 Amplified reality to overcome introspection
Three notions of ambient media have been pointed out in  by W. Andrew Schloss and D. Stammen and applied in their artistic installations. The space for ambient media is our surrounding—our public or private environment, that are stimulating our senses. Thus, we are exposed to content continuously and in certain application areas the exposure is unconscious. This allows no direct interaction between the consumer and the medium as used to in typical model-view-controller applications dominating today’s digital media world. In existing media, such as e.g. the Web, we get immersed into the story world (the synthetic world) consciously and the temporary ‘real’ world is the digital one—thus we are consciously isolated in the synthetic world. However, while walking through public environments, we tend to ignore stimuli as we are accustomed to familiar places on our daily travel to work. Introspection is dominating in this case (introspection vs. isolation)—eventually we can speak even of information overload based on the isolation through technology (e.g. discussed in ). They present a soundscape linked to objects and conditions of the natural environment within artistic installation. The soundscape creates amplified reality (see also ) and allows people to overcome introspection based on the general tendency to “turn out nature and [with the use of a sonification techniques] try to tune them back in [, ...] as it becomes progressively more important for people to have physical places to go” .
7 Media content and form: media & arts perspective
understanding of contemporary media environments
traditional media environments and ambient media
aesthetics of ambient media
extension of existing media environments
enabling collaboration and communication of feeling a shared space
7.1 Understanding of contemporary media environments
Before discussing about advanced ambient media forms, we have to understand contemporary media culture and its potential in relation to ambient media—and let it be ambient or not, the media content is still a story. The paper by S.-M. Chung, P.-C. Ho, and C.-H. Huang  presents the results of a study about the visual and aural expressions that may incur interaction in an interactive media software. The story they are relating to is the concept of Seven Deadly Sins by Saint Thomas Aquinas, AD. 1225–1274. Background music and colors are used to transfer the mood—thus, anger is e.g. expressed by the adequate background music. The developed interactive software shows an interesting aspect of ambient media, where media have been interpreted by a human in relation to the mood of the people. Therefore moods are the semantic model of the application—by matching content elements, music and colors and their composition they affect the experience of the consumer. In current media environments annotations and the combination of content objects are done manually. However, models of composition of visual and auditory media objects to transfer mood are as well major concern in semantic ambient media environments. One major role will be played by the automated or semi-automated creation of semantics—a meaning for the consumer. Assuming this as a starting point, we can show a few attempts by artists attempting to realize application scenarios for ambient media environments. These are listed mostly in the following sections of this article.
7.2 Automated aggregation of content
In ambient media systems, content is not delivered in a push modality, and not accessed in a pull modality. In ambient systems, the system aggregates the content on behalf of the consumer. A good example has been contributed by O. Motoyuki, M. Shunichi, O. Kanako, and N. Yuichi . Their research work focuses on how to improve the quality of ambient content. It is a fact that more and more sensor networks are embedded throughout our natural environment. However the main problem is to obtain knowledge, meaning, structure, and semantics out of the data. In very simple words, how can ambient content—thus hours of recorded video cameras or other sensor data—be turned into enjoyable content. In existing systems, data is difficult and boring to interpret, but the authors propose a digital assistant to make recorded video content more understandable and enjoyable. Thus, the agent acts as smart device for the automated aggregation of content. In an experimental setting during a cooking show, the artificial agent could influence the behaviour of the TV show presenter positively. The experiments show, that a natural and smart helper can be created in advanced use scenarios—the system as aggregator of content, thus as virtual content producer.
7.3 Preparing the traditional media environments for ambient media
Broadcast media are busy with the introduction of HD production work-flows these days, with compliant consumer electronic devices readily available for a few years now. Together with 5.1 surround sound home cinema installations can create an atmosphere that can compare with classic cinema. A feeling is that there is not much left to add: If the viewer can watch a movie in high definition sound and image, fully immersing him into the story flow and letting him forget that he is sitting on a sofa, then the mission is accomplished. There is little attention left to direct towards supporting media such as ambient light in the peripheral area of the visual field, so whatever can be added to the viewing experience can only work subtle.
It is unlikely that broadcast TV will catch up fast with producing ambient media enriched content any soon. HD quality is a huge step to be taken and as expensive innovations come along with the equipment re-investment cycles, we should not expect ambient media in homes earlier than in 5 years time. And not to forget that the next important step might already be 3D TV. The situation in the movie industry is similar, full digital production work-flows are getting common and digital cinema with 4 K projectors will offer unprecedented image quality as next step of cinema evolution. So far cinema has been the paradigm for TV technology innovation, but with ambient media, the development may be uncoupled. Taking ambient lighting as an obvious and simple example, lighting in a big cinema with hundreds of viewers has different constraints than lighting a living-room with just a few people viewing. The key question is whether this is feasible and wanted in cinema at all. If so, the movie industry will sooner or later start enriching their movies with ambient content and then those movies will as well be a driver for TV-related ambient media. If not, production of ambient media content is unlikely to start from within the tightly-budgeted world of TV productions. A factor that could foster the production of ambient media content by the movie industry is the shift of revenues from cinema to DVD, BluRay, and Online Movie Rental market.
Apart from hand-crafted content, ambient content can as well be extrapolated from the original media. This can be done within the end devices and is likely to be introduced into future home cinema systems to make a distinction on the market. So far, Philipps Ambilight is the only technology already available to the customer. LCD display technology is about to reach the point where innovation in terms of visible picture improvement is not possible anymore. 200 Hz technology is an example for a technical improvement that has exceeded the resolution of human perception by a rough factor of two. In such situations only real innovation can give a manufacturer an edge on the competitors. Such innovations could be 3D TV or ambient media. It will be a matter of a cost-benefit calculation, which comes first. Some ambient media concepts can offer astonishing improvements in user experience for a cheap price. Given that authored ambient media content is not likely to be available for the next few years to come as explained earlier, extrapolated ambient content driven by the consumer electronics industry seems to be the near future for ambient media.
8 Collaboration, interactivity, participatory, and experience: consumer perspective
systematic creation of consumer experiences
semantic ambient media supporting daily life situations
implicit collaboration and communication
collaboration, interaction and participation
new methods for collaboration and participation
Creating content for ambience is not new in entertainment industry as they always attempted to embed their creations into a well-designed ambience. Be it cinemas, variety theatres or Rome’s Colosseum—they all were designed to create an atmosphere that greatly enhances the customer’s experience. With the rise of electronic media, the entertainment industry lost control of the reception environment. First consumers listened to radio and watched TV in their living-room, nowadays they just do so anywhere they want. Ambient media has the mission to re-conquer the reception environment. Naturally consumers will be reluctant to give away control over the equipment. They may be only convinced by great ambient media applications to do so.
8.1 Ambient media supporting daily life situations
Smart technology can enable and support the consumer in daily life situations. Very simple examples are GPS, location based services, and the smart fridge. However, one more specific scenario is ambient assisted living for the elderly. V. Fuchsberger addresses this challenge and describes how the elderly can be supported to stay longer in their homes by the utilization of ambient technology . The general aim is to support the elderly in daily situations with essential technology to maintain their activities in their accommodated home environments. When talking about the elderly, there are many obstacles. One of the major one is, how the elderly will deal with emerging new technology, and if they are accepting this technology. One example of new technology stated in  is IPTV as possible medium for education of the elderly in the use of latest technology.
8.2 Systematic creation of consumer experiences
An interesting aspect in the creation of consumer experience is the relation between stories and experiences. It is a fact, that certain stories trigger experiences in consumers. N. Sharda  suggest a concept for experience on-demand as a recursive five level approach to derive user experience from emotions, feelings & thoughts, behaviour & action down to system & component level. His specific contribution lays in the attempt to develop a reproducible model for consumer experience independent from time and place. His theory is based on the opinion of the anthropologist Levi-Strauss, who stated, that the human brain uses narratives to store and make sense of one’s life events . Thus the reproduction of experiences can be based on stored narratives to provoke certain emotions in people. Sharda proposes the Movement Oriented Design (MOD) model to create well narrated multi-medial stories especially developed for e-learning environments. In MODs, the stories are created from smaller units with a small but complete story with a beginning, middle, and end. Each story unit triggers emotions and interested by the consumer. A plot is created to evoke a specific user experience in an ambient media environment. However, the challenge is, which formula can be applied to find the correct relation between story, emotions one story triggers, and the plot.
8.3 Implicit collaboration and communication
Another contribution by J. Frank, T. Lidy, E. Peiszer, R. Genswaider, and A. Rauber  asks the key-question in ambient media systems—Where is the display and how can ambient content be projected and stimulate implicit collaboration? Visual content rendering is bound to discrete display technology—however, visual displays require our full attention and contradict the nature of ambient media of being embedded throughout our natural environment. They argue, that audio is perfectly suited to fit for ambient media—audio is ambient! They introduce the MusicSOM Café, where guests can choose different music by moving between the tables of the Café. Music sources can be rendered by placing music sources throughout the natural environment and the prediction of the sound distribution. Nevertheless, the system goes also one step further—sound is not only rendered into the natural environment—the sound is also adapted to different music tastes. Collaboration is basically based on the implicit data collection to catalyze social processes—people with a similar music tastes are very likely to have similar interests in general. Their system is aimed at fostering social communication and social contacts. Emotional sound design—or even better—social sound design is a huge and challenging topic per-se, however, adding audio to locations add a new dimension to the impact of ambient media on society and collaborative processes.
8.4 Personalization—Reducing the information overload
In semantic ambient media environments, it is important to compose the most valuable service for a specific user at the right moment. M. Hossain, P. Atrey, and A. Saddik  work focuses on resolving the question, how much gain a consumer would get from a particular service, if this content would be presented to him. The approach is based on a context profile, holding a history of observed consumer activities. The approach aggregates several different user data sources (similar to the approach of P2 [24, 33]), and recommends the consumer with the correctly prioritized service. This technique is a good example for reducing the information overload for the consumer by providing him with a personalized environment—smart system adaptation to changing consumer needs are one major property of ambient systems.
8.5 Domain specific collaboration
Artistic expression and the use of technology have been often discussed (e.g. in motion pictures when moving from analogue film to high-definition). L. Mandilian  evaluates ambient technology in dance performances. The author’s aim was to evaluate how choreographers explore ambient technology for the use in performances. The paper is an extraordinary example of two different issues: first, the artistic piece as such deals with information overload presented via a dance performance entitled Information Overload premiered in 2007; second, a dance performance integrating digital media and audience responses. Ambient media embedded as part of a dance performance, where the ambience of the theatre hall and the performance as such become an interactive environment responding to an interactive audience. The paper helps to understand what actually interactive ambient media could be in the context of artistic performances. Seeing audience participation as collaboration to achieve a dance piece, the dance performance nicely presents a domain specific collaboration from the artistic perspective.
9 Workshop results and discussion
The workshop was divided into two blocks—presentation block in the morning, and discussion block in the afternoon. During the presentation block the authors had a chance to present their works. In the afternoon block—the discussion block, the authors discussed within their group and presented the results at the late afternoon to the audience.
The outcome where remarkable, and group 1—the ‘Spirituals’—developed the XAura—as an aura consisting of a n-tuple defining the important aspects and relation of user, system, and environment. The aura consists of people, devices, and environmental objects.
As a summary, to combine existing trends in the field of semantic ambient media is a rather huge challenge. Within the workshop (see Table 1), we were able to focus on many aspects on the various questions we stated before the workshop. However, the answers seem to lead to more and more different other aspects of ambient media, and let it seem that a framework for semantic ambient media is more and more difficult to develop. The multidisciplinary nature makes it clear, that much more research work is required. Especially as semantic ambient media are just on the brink of emergence. Only a mix between disciplines and different viewpoints will allow us to create a novel approach towards semantic ambient media.
At first we would like to thank all the SAME 2008 contributors and authors! Several of them are referenced with their valuable contributions to this article. Without their lively discussion and participation in the workshop this article would have not been possible!!! This work was supported by the Academy of Finland, project No. 213462 (Finnish Centre of Excellence Program (2006–2011). We would like to thank our numerous friends, colleagues, and especially the contributors for our workshop.
- 1.Anderson C (2004) The long tail. In: Wired magazine, vol 12. San Francisco, pp 170–177Google Scholar
- 2.Bergamaschi S, Castano S, Vincini M (1999) Semantic integration of semistructured and structured data sources. In: SIGMOD Rec. 28, 1Google Scholar
- 3.Berners-Lee T, Hendler J, Lassila O (2001) The semantic web. In Scientific American. vol May, pp 34–43Google Scholar
- 4.Bischoff K, Firan CS, Nejdl W, Paiu R (2008) Can all tags be used for search? In: Proceeding of the 17th ACM Conference on information and Knowledge Management. CIKM ‘08. ACM, New York, NYGoogle Scholar
- 5.Bowman S, Willis C (2003) We media: How audiences are shaping the future of news and information. The Media Center at the American Press InstituteGoogle Scholar
- 6.Chung S-M, Ho P-C, Huang C-H (2008) A case study for a multimedia program. In: Proceeding of the 1st ACM international workshop on Semantic ambient media experiences. ACM, Vancouver, British Columbia, CanadaGoogle Scholar
- 7.Codognet P, Nouno G (2008) Red light spotters: Images-driven sound and rhythm landscape. In: Proceeding of the 1st ACM international workshop on Semantic ambient media experiences. ACM, Vancouver, British Columbia, CanadaGoogle Scholar
- 8.Cunningham H, Maynard D, Bontcheva K, Tablan V (2002) GATE: A framework and graphical development environment for robust NLP tools and applications. In: Proceedings of the 40th Anniversary Meeting of the Association for Computational Linguistics (ACL ‘02). PhiladelphiaGoogle Scholar
- 9.Dowman M, Tablan V, Cunningham H, Popov B (2005) Web-assisted annotation, semantic indexing and search of television and radio news. In: 14th International World Wide Web Conference. Chiba, JapanGoogle Scholar
- 11.Frank J, Lidy T, Peiszer E, Genswaider R, Rauber A (2008) Ambient music experience in real and virtual worlds using audio similarity. In: Proceeding of the 1st ACM international workshop on Semantic ambient media experiences. ACM, Vancouver, British Columbia, CanadaGoogle Scholar
- 12.Fuchsberger V (2008) Ambient assisted living: elderly people’s needs and how to face them. In: Proceeding of the 1st ACM international workshop on Semantic ambient media experiences. ACM, Vancouver, British Columbia, CanadaGoogle Scholar
- 14.Hossain MA, Atrey PK, Saddik AE (2008) Dynamic gain estimation in ambient media services. In: Proceeding of the 1st ACM international workshop on Semantic ambient media experiences. ACM, Vancouver, British Columbia, CanadaGoogle Scholar
- 15.ISTAG (2001) Scenarios for ambient intelligence in 2010—Final Report (Feb. 2001). http://www.cordis.lu/ist/istag.htm
- 16.ISTAG (2003) Ambient intelligence: From vision to reality. European Union: IST Advisory Group, draft reportGoogle Scholar
- 17.Jenkins H (2006) Convergence culture: where old and new media collide. NYU, New YorkGoogle Scholar
- 18.Lévi-Strauss C (1995) Myth and meaning: Cracking the code of culture. Schocken Books, New YorkGoogle Scholar
- 19.Lugmayr A (2006) The TICSP workshop on ambient multimedia and home entertainment at the EuroITV 2006. In: TICSP #33. Tampere University of Technology (TUT), Athens, GreeceGoogle Scholar
- 20.Lugmayr A (2006) The future is ‘ambient’. In: Creutzburg R, Takala JH, Chen CW (Eds) Proceedings of SPIE, vol 6074, 607403 multimedia on mobile devices II, vol 6074. SPIE, San JoseGoogle Scholar
- 21.Lugmayr A (2007) Ambience, ambience, ambience—what are ambient media? In: Interactive TV: A shared experience, TISCP Adjunct Proceedings of EuroITV 2007, Amsterdam, http://www.cs.tut.fi/~lartur/euroitv07_ajp/main.htm
- 22.Lugmayr A, Pohl A, Mühlhäuser M, Negru D, Kallenbach J, Köbler F, Niiranen S (2006) Ambient media and home entertainment (Workshop Review). In: Proceedings of the 4th Euro iTV Conference. ELTRUN/AUEB, Athens, GreeceGoogle Scholar
- 23.Lugmayr A, Risse T, Stockleben B, Kaario J, Laurila K (2008) Semantic ambient media experiences same 2008 pre-workshop review (NAMU series). In: Proceeding of the 1st ACM international workshop on Semantic ambient media experiences. ACM, Vancouver, British Columbia, CanadaGoogle Scholar
- 24.Lugmayr A, Reymann S, Kemper S, Dorsch T, Roman P (2008) Bits of personality everywhere: Implicit user-generated content in the age of ambient media. In: Parallel and distributed processing with applications, 2008. ISPA ‘08. International Symposium on, pp 516–521Google Scholar
- 25.Mandilian LE, Diefenbach P, Kim Y (2008) Information overload: A collaborative dance performance. In: Proceeding of the 1st ACM international workshop on Semantic ambient media experiences. ACM, Vancouver, British Columbia, CanadaGoogle Scholar
- 26.Ozeki M, Maeda S, Obata K, Nakamura Y (2008) Virtual assistant: An artificial agent for enhancing content acquisition: How ambient media elicit information from humans. In: Proceeding of the 1st ACM international workshop on Semantic ambient media experiences. ACM, Vancouver, British Columbia, CanadaGoogle Scholar
- 28.Rakkolainen I (2008) Mid-air displays enabling novel user interfaces. In: Proceeding of the 1st ACM international workshop on Semantic ambient media experiences. ACM, Vancouver, British Columbia, CanadaGoogle Scholar
- 29.Rakkolainen I, Lugmayr A (2007) Immaterial display for interactive advertisements. In: ACM conference on advances in computer entertainment technology. Salzburg, Austria, pp 95–98Google Scholar
- 30.Rakkolainen I, Palovuori K (2002) A walk-thru screen. In: IS&T/SPIE Electronic Imaging 2002, Conference on Projection Displays VIII. San Jose, CA, USAGoogle Scholar
- 31.Rakkolainen I, Palovuori K (2004) Interactive digital fogscreen. In: Proceedings of the third Nordic conference on Human-computer interaction. ACM, Tampere, FinlandGoogle Scholar
- 32.Resource Description Framework (RDF) (1999) W3C 1999, http://www.w3.org/TR/PR-rdf-syntax
- 33.Reymann S, Bruns V, Lugmayr A (2007) P2—Portable personality a middleware solution for smart user profile management and distribution. In: Interactive TV: a shared experience, TISCP Adjunct Proceedings of EuroITV 2007, AmsterdamGoogle Scholar
- 34.Riva G, Vatalaro F, Davide F, Alcaniz M (2005) Ambient intelligence. IOS, http://ambientintelligence.org
- 35.Schloss WA, Dale S (2008) Ambient media in public spaces. In: Proceeding of the 1st ACM international workshop on Semantic ambient media experiences. ACM, Vancouver, British Columbia, CanadaGoogle Scholar
- 36.Schütze H (1998) Automatic word sense discrimination. Computational linguistics 24:97–123Google Scholar
- 37.Sharda N (2008) Creating ambient multimedia experience on-demand: opportunities, challenges and research models. In: Proceeding of the 1st ACM international workshop on Semantic ambient media experiences. ACM, Vancouver, British Columbia, CanadaGoogle Scholar
- 38.Simon R, David SA, Artur L (2008) Personalized social networking: An applied scenario in a portable personality environment. In: Proceedings of the 12th international conference on Entertainment and media in the ubiquitous era. ACM, Tampere, FinlandGoogle Scholar
- 39.Singh VK, Pirsiavash H, Rishabh I, Jain R (2008) Towards environment-to-environment (E2E) multimedia communication systems. In: Proceeding of the 1st ACM international workshop on Semantic ambient media experiences. ACM, Vancouver, British Columbia, CanadaGoogle Scholar
- 40.Smith MK, Welty C, Guinness DLM (2004) OWL—Web ontology language guide. W3C 2004Google Scholar
- 42.Streitz N, Prante T, Müller-Tomfelde C, Tandler P, Magerkurth C (2002) Roomware©: the second generation. In: In CHI ‘02 extended abstracts on human factors in computing systems. CHI ‘02. ACM, New York, NYGoogle Scholar
- 43.Streitz NA, Röcker C, Prante T, Alphen DV, Stenzel R, Magerkurth C (2005) Designing smart artifacts for smart environments. IEEE Computer, vol March, pp 41–49Google Scholar
- 44.Szu-Ming C, Pei-Chi H, Chun-Hsiung H (2008) A case study for a multimedia program. In: Proceeding of the 1st ACM international workshop on Semantic ambient media experiences. ACM, Vancouver, British Columbia, CanadaGoogle Scholar
- 45.Weiser M (1994) The world is not a desktop. Interactions, pp 7–8Google Scholar
- 46.APIS—Advanced Personalization in Information Services, http://www.l3s.de/apis/
- 47.AR Toolkit, http://www.hitl.washington.edu/artoolkit/
- 48.Fring, http://www.fring.com
- 49.Geocaching, http://www.geocaching.com/
- 50.Raftery T, Creative ways of reducing your carbon footprint. http://www.slideshare.net/TomRaftery/reducing-a-data-centers-carbon-footprint/