Keywords

1 Introduction

Augmented reality (AR) applications have recently drawn a considerable interest, triggered by the announcement and public discussion of devices like Google GlassFootnote 1. While this discussion is mostly centered around the hardware aspects of AR, the far more interesting questions arise in the area of applications. These applications are driven by novel interaction models, going beyond the explicit approaches used on computers and smartphones, since such interaction is very cumbersome on AR devices. Instead, information has to be delivered proactively in a contextualized way, necessitating a different approach to data management and presentation. In this demo, we show such an application and its accompanying software in the form of an interactive conference guide. The AR device provides the participants of a conference with relevant background information about presenters, events or topics around the conference based on the current context, e.g., publications of a presenter who shares the same research interests with a participant. This application is developed within the Durchblick projectFootnote 2, which extends an already proven conference information system (conference explorerFootnote 3) based on semantic technology. It relies on personal information such as social networks or public data such as conference schedules or publications collected there, combines it with contextual information and applies state-of-the-art recommendation and event processing techniques to tailor the information presented on the AR device in a rapid fashion.

Existing conference support applications like Conference4MeFootnote 4 are smartphone-based and provide a static display of conference information with limited personalization, thus requiring explicit user action. Utilizing context information to tailor information has been studied extensively, e.g. for digital libraries [2] or information streams [1], but has rarely been developed to full systems. Employing semantic technology in AR settings has only recently gained attention, such as the semantic event processing in the ARTSENSE project [4].

2 Demonstration Scenario

In the following we present a scenario as it will be demonstrated based on actual conference metadata at the conferenceFootnote 5: Consider a researcher (called ‘Sarah’) who plans to attend the ESWC’14 conference. Like most other attendants, her goals for a conference visit are twofold: (1) learning more about relevant advancement in the fields and (2) connecting to other researchers in her domain. For this purpose, she will get support from the context-aware conference support system in Durchblick. This system employs Google Glass to deliver its services enabling Sarah to gain the maximum benefit from her visit in various ways, combining user profiles and contextual knowledge: According to her research interests obtained explicitly from her LinkedIn profile and implicitly from the publication registry, Durchblick recommends an initial program for Sarah’s visit to the conference. The program is tailored towards conference events that match her preferences and interests. Additionally, she can build her own personal schedule by directly selecting available events, rejecting or accepting the recommended events, initially or during the conference. Such interactions (feedback) on the recommended events along with user’s activities on the conference are employed to enhance the recommended program over time and provide more personalized recommendations. Figure 1 shows an example of a notification of an upcoming event and possible interactions. While attending talks and meetings, ‘Sarah’ is able to see the background information of the presenter such as previous publications or research topics and learn more information about the presentation topic on her AR devices. Durchblick explores the background information from social network and publication registries and uses context information such as schedules or location to automatically select the relevant parts. Once ‘Sarah’ takes a picture and shares it via the Glass, Durchblick provides annotation information related to the picture and shares it with other participants who may want to search for photos related to the conference or its specific sessions. Finally, Sarah gets all important announcements made by conference organizers via Durchblick, like rescheduling events or announcing a delay on the plan for the conference dinner.

Fig. 1.
figure 1

Use Case Scenario: UI flow and screen shots

Full size image

3 Architecture

Durchblick tackles the challenges of AR applications in two ways: first, it perceives and models the user context for a better understanding of her needs, and second, it utilizes the linked data integrated from different resources: social networks; publication registries; and conference meta data. Both aspects help providing a rich platform for delivering the suitable information proactively. As shown in Fig. 2, the user interface layer (the top layer) with the AR device, plays the main role not only in delivering the requested information to the user, but also in capturing the user context. The second layer (Durchblick application server) however, takes care of modelling that context properly. On the other hand, Durchblick business logic is depicted in this layer through the collaboration of different components appearing here (Recommender, Event processing, and context-based query processing components). Integrated linked data is made available by the means of the two layers at the bottom of the hierarchy. Those layers are responsible for collecting data from the original sources in various formats and integrating them in the form of a unified RDF data graph. In this section we will present the user context and delve deeper into the second layer giving more details about its components.

Fig. 2.
figure 2

Durchblick architecture

Full size image

User context: In Durchblick we define two categories that characterize the user context: Situational context and historical context. The situational context covers user location, current time, social context, and user preferences. The history includes previous actions, events attended, and recommendations reacted to.

In order to model the user context, we reused several vocabularies from different ontologies. The situation pattern from the DUL ontologyFootnote 6 is employed as a context holder that models user’s time, location, and the current attended event. For representing user preferences, we resort to the weighted interests ontologyFootnote 7.

Event Processing: The Event processing component handles events delegated by the user interface layer up on different user actions. Those actions may be explicitly triggered by the user as a request for a specific service (background information, recommended events,...) or derived implicitly upon a detected change in the user context. The event processing component models those events and forwards a contextualized user object to the responsible component. Specifically, in the case of Google Glass, events are received via the Mirror API, which provides a subscription mechanism for events detected on the device, such as change in location or a particular user action.

Recommender: Recommendations are based on the context model of the user. Here, two different kinds of context for the user profile are distinguished:

(1) Long-term context that depicts the profile of the user which does not change during the conference and is known from external data sources, such as research interests listed by the user (on her LinkedIn profile for example) and learnedFootnote 8 publication records, and (2) short-term context that is defined by the history of actions since the beginning of the conference (e.g. the likes/dislikes of a user, attended talks/presentations, etc.) in addition to the situational context, which is detected by the event handling component. Durchblick provides a schedule recommendation that is a program of different talks taking place during the conference. The recommender component applies a content based approach [3] modelling the user profile as a vector of key-value pairs, where domain specific keywords that appear in user’s preferences are the keys while the values express how much these keywords are important to the user. Talk events on the other hand are modelled in the same approach using their keywords and the corresponding normalized frequencies as score values.

Context-based Query Processing: This component is responsible for inferring information objects that are relevant in the current user context. The information objects can be, for example, recommendations, notifications or announcements. We resort to SPIN (SPARQL Inferencing Notation) for encoding the business logic to reason about context. Specifically, the rules are used to (1) detect higher level situation descriptions (e.g. attending a session, interacting with a person) based on low level events, (2) inferring relevant information objects based on the current situation, and (3) establishing relationships between objects in a particular situation (e.g. to annotate a shared picture with the current user context). Upon context changes, the rules are evaluated and the inferred relevant information objects are sent pro-actively to the user. In the case of Glass, we again make use of the Mirror API to display timeline cards for the information objects.

4 Conclusion

In this demo, we provide insights how upcoming AR devices greatly benefit from semantic data management. We build an architecture that puts contextualized query processing, event processing and recommendations on top of semantic data and delivers relevant information proactively. Currently, we employ early prototypes of these components; we plan to improve them in many aspects: The interaction context and recommendations needs further investigation, both from the conceptual side (recommendations under constraints, parametrized recommendations) and the system design side (realtime results vs. context flexibility). Similarly, the interaction of visualization and context-based query processing needs to be studied in more detail.