Keywords

1 Introduction

With the advent of more powerful mobile computing power, AR has received increasing attention in industry and in the consumer market. New platforms and new devices create an expanding design space of AR systems [3].

While others have used AR to support general user centered design processes [4, 14], we are using virtual reality (VR) to support the user centered design of AR applications specifically. DART [9], as an early approach for AR prototyping, was very successful at its time, providing a toolchain enabling designers to create working prototypes as a result. This, however, brought about a stronger focus on the technology side in the design process, restricting the designer to what was possible with the technologies provided with DART. With ExProtoVAR (Experience Prototypes in VR for AR applications) we want to abstract away from specific hardware solutions and software implementations and focus on the design of the user interaction with the AR system and the experience of the users.

Important aspects, that have to be considered when designing for AR are: (1) the context (spatially) of the situation, (2) the dynamics of the situation (temporally), (3) the user’s whole body in the loop, and (4) the features of the AR device. While (1) and (2) depend on the application scenario, (3) depends on the user, the task and her experience with AR. The choice of the AR device (4) may be part of the design or fixed, however it will always influence (3).

The situational aspects (1+2) will often be very specific, e.g. when designing for AR maintenance of a certain machine, and they might cover several locations and states (representing task progress). During design and prototyping, it is thus essential for the designers to have a realistic representation to work with. The choice of the AR device (4) will, at least in industrial settings, depend on the interaction between 1, 2 and 3 to achieve a certain performance goal.

We try to address these challenges with ExProtoVAR: a lightweight tool to create interactive virtual prototypes of AR applications. The central idea is to cover situations (1+2) using sequences of 360\(^\circ \) panoramas. In contrast to other tools, ExProtoVAR is primarily used in VR, giving the designer an immersive editor at hand, with which virtual tours and state transitions can easily be defined in close interaction with the users. Areas of interest can be defined and sketches or HTML prototypes of AR content can be integrated to design the AR application. Different presentation modes, such as in-view (attached to the screen) and in-situ (attached to the environment), can be defined (see Fig. 3). Different AR devices (iPhone, Samsung Tablet, Google Glass, Microsoft HoloLens) are simulated (4) to support an informed choice. In evaluation mode, users can immerse into the prototype, try it out and use audio and text annotations to comment on certain features.

2 Related Work

The topic of AR is only scarcely addressed by the design community (e.g. [6]) and little to none design patterns and principles addressing AR applications exist. This makes it even more important to support the early phases of prototyping, as designers and users can only rely on minimal shared experiences.

The idea to use panoramic images or videos for prototyping AR has been introduced before (e.g. [2]). Others have used 3D simulations in immersive virtual reality [1] with expensive hardware for AR prototyping. In a sense, we are using VR to simulate AR. This is a quite successful approach applied by the AR community to explore designs for AR hardware and basic mechanisms (e.g. displays) [7, 8, 11, 12, 15, 16].

ExProtoVAR concentrates on holistic AR experiences produced with a low technical barrier and low developmental costs. Regarding the distinction between low-fidelity and high-fidelity prototypes (see e.g. [13] in the mobile context), our work ranges in the medium-fidelity range. One way it extends upon PARnorama [2] is, that it immerses the user into the situation rather than showing only a snippet of the situation on the screen of the AR device.

3 Motivation and Approach

To build prototypes for AR applications a tool is needed to support users, designers and developers in different steps of a prototyping cycle. In the ProFI project [10], we orient ourselves on the double diamond model [5]. This model divides the design processes into four distinct phases called Discover, Define, Develop and Deliver. The prototyping process for AR can be seen as a special case leading to several challenges on the way to a product (see Fig. 1).

Fig. 1.
figure 1

The double diamond model for prototyping processes adapted to the design of AR solutions supported by ExProtoVAR.

Full size image

  • Discover. In the first phase, designers try to look at the world in a fresh way, notice new things and gather insights. Here it is important to make customers acquainted with the possibilities of AR. The first phase is characterized by a creative divergent thinking and a tool is needed to make AR concepts come alive, providing broad ideas on what could be possible. As AR is very dependent on the situation, access to the environment the application is planned for, is crucial. A tool is required to meet and immerse into the environment and discuss possible augmentation ideas.

  • Define. In the second phase, designers need to determine the best and feasible approaches identified in the discover phase. The goal here is to develop a clear creative brief that frames the fundamental design challenge. To this end, a tool can be of great benefit which enables the designer to create an interactive prototype in an easy way and within a short period of time. The prototype of this phase can function to convey the central ideas to the designers in the next phase and can be seen as a representation of the convergent thinking process of this phase.

  • Develop. Several prototypes are created, tested and iterated in the develop phase, targeting at a concrete prototype. This process of trial and error helps designers of the user interface and the AR markers to improve and refine their ideas. Additionally, technical adaptations and solutions need to be developed and tested as well as usability tests need to be performed. As different people are working an different aspects e.g. designers on interface design and technicians on which device to use, these aspects always need to be integrated in one prototype, so that side effects can be accounted for. A shared tool is needed which is able to integrate the different prototypes and ideas.

  • Deliver. In the last phase, the resulting project (a product, service or environment) is finalized, produced and launched. A prototype with notes and additional explanations can help to serve as benchmark for the solution offering a tool representing the different ideas and stages of the prototype process.

4 ExProtoVAR

With ExProtoVAR, we present a lightweight tool to create interactive virtual prototypes of AR applications. The central idea is to cover context situations using 360\(^\circ \) panoramas. This serves two purposes; on the one hand, it helps the user to immerse in the situation the application is developed for and thus gets a feeling for using the application in the relevant context. On the other hand, the environment can be used to identify the areas later used as AR markers. Consumer panorama cameras are sufficient, small and easy to use. We wanted to create a tool with which a situation can be modeled in less than an hour and with which it is easy to deal with several alternative images. Also, the VR environment simulating the AR scenario is not supposed to be too complex, as its purpose is to serve as an early prototype. Figure 2 displays the interface and two examples of content being displayed during an ExProtoVAR session.

Fig. 2.
figure 2

Example shots taken from within an immersive session in ExProtoVAR. TL: Samsung GearVR headset and controller; TR: Main menu of ExProtoVAR; LL: Samsung Gear 360 panorama camera; LC: In-situ information in a museum. LR: In-view presentation of product information in the supermarket.

Full size image

The main part of prototyping happens immersed in VR. Only sketches which are to be used as AR content have to be created in a traditional 2D way. The designer can easily structure the experience by putting panorama pictures and AR content (e.g. images, HTML pages) in folders on the smartphone. To create the VR experience, the designer puts on the VR headset (Samsung GearVR, see Fig. 2) and uses the controller to define the logical, temporal and spatial aspects of the situation by use of situation-links. She then marks areas in the panoramas and links them to the prepared AR content (we call this event-links). Here, she can decide whether to present the content in-view or in-situ (see Fig. 3). For the latter, the anchoring of the content in the environment can be defined as well.

Integrating Devices. Different simulated devices can be used to experience the AR content, e.g. a Samsung Galaxy phone, an Apple iPad, an Apple iPhone, a Google Glass or a Microsoft HoloLens. They are simulated with respect to their size, resolution and display type. Handheld devices are used in the virtual scene by moving the controller, glasses automatically follow the user’s head movements. By this ends, the user is able to look around while scanning the environment for augmentations and develop a feeling for the behavior of the different devices.

Defining Device Interactions. ExProtoVAR supports not only a basic scanning for AR content, but also the modeling of interactive screens. For this, the device can be brought up closer to the user by the press of a button. The device is then filling almost all of the screen and detached from hand movements. The controller now instead operates a virtual cursor which can be used to simulate touch events on the AR device’s screen. If the AR content is an interactive HTML page, e.g. a click dummy, HTML events can be triggered as usual and the content will interactively change. A fake “situation” protocol (scene://) is used to allow the HTML content to trigger situation transitions to new panoramas, e.g. to simulate actions that control a machine and evoke new states.

Annotations. After a prototype has been created with the ExProtoVAR tool, it can be reviewed by users. To enable the user to provide context specific feedback, it is possible to mark areas in the prototype and leave notes and audio notes. The notes will be displayed exactly in the chosen situation at the marked position and can be edited or complemented in subsequent sessions. This supports the feedback cycle and eases the communication about context dependent issues.

Videotaping an Interaction. An additional feature to provide a compressed version of a prototype consists of videotaping an interaction covering all relevant interactions with the AR application in the virtual world. This can then serve as a basis for the developmental phase or can be used to present the prototype to several people at once not using the VR equipment.

Fig. 3.
figure 3

Different visualizations of in-situ (left) and in-view (right) content.

Full size image

5 Example and Evaluation.

ExProtoVAR itself has been subject to iterative prototyping in a user centered design manner and insights have been gained by 8 expert interviews in review sessions. Personas have been designed and example design tasks to decide upon were given to the experts. The experts agreed that ExProtoVAR immersed them into the applications environment and made them feel involved. This helped them to identify with the decision process. In the future, ExProtoVAR will be used in customer projects for further evaluation.

Fig. 4.
figure 4

Using ExProtoVAR to prototype an AR exhibition guide for a museum

Full size image

Figure 4 shows an example of the design phase of a customer looking for new ideas to convey information in her museum (1). After contacting the AR company (2), the customer is presented with impressions of AR technology. She gets to see a collection of relevant prototypes with the ExProtoVAR tool to demonstrate the potentials of using AR (3). As a preparation for the first prototype, panorama pictures of the museums exhibition rooms are taken (4) and used to create an individual panorama prototype (5). This prototype serves as a basis for designers and developers to prototype an application fulfilling the requirements and desires of the customer (6). To this end, different interface and content designs are prototyped to convey the information in the best possible way. To raise funds for the museums AR application, the customer presents a video of the prototype to the city council (7). As a last step, the application is subject to refinement processes before being released (8).

6 Conclusion and Future Work

Designing contextualized (AR) applications requires a careful consideration of the situation. We have presented ExProtoVAR, which combines VR and panorama imaging technologies to immerse designers and users in the situation while defining and evaluating different interaction designs. Simulations of AR devices and different visualization styles allow for a functional evaluation, while the immersion supports an emotional evaluation at the same time.