Introduction

In the Otherland saga, Tad Williams (1996, 1998, 1999, 2001) describes a future world with a widespread availability of full-immersion Virtual Reality (VR) installations which allow people to access an online world, called simply ‘the Net’. Within the Net, a group of people aim to achieve immortality. In his novel Rainbows End (Vinge 2007) Vernor Vinge describes how the main character, Robert Gu, is slowly recovering from Alzheimer’s disease and adapts to a changed world in which almost every object is networked and the use of Augmented Reality (AR) is normal. Humans experience AR by wearing smart clothes and contact lenses that can overlay the physical environment with computer graphics. In this future world, AR technology is used for various purposes, such as large-scale commercial gaming, supporting maintenance workers with blueprints of machines or buildings, communication with virtual avatars, and medical diagnosis.

AR allow users to see the real world, with virtual objects superimposed upon or composited with their real environment (Azuma 1997). Here the virtual objects are computer graphics that exist in essence or effect, but not formally or actually (Milgram and Kishino 1994). AR systems are not limited to use of head-mounted devices (HMD), but AR technology mainly has to combine real and virtual objects, to be interactive in real-time, and to register virtual objects within the real 3D environment (Azuma 1997).

Williams and Vinge forecast a vision for the future that current research on collaboration in AR is addressing. Several years from now, technology will provide an infrastructure for physical and virtual connectivity just as described in Rainbows End. Everyday objects will be connected and be able to provide and exchange information. Instead of only overlaying the physical environment with computer graphics, future AR systems will move closer to the ultimate display that Ivan Sutherland envisioned as “a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal.” (Sutherland 1965). Such a display would allow for holistic embodied experiences addressing all of the human senses, such as sound, smell, taste and touch, as envisioned in Otherland. Early science fiction motion pictures, e.g., Star Wars (Lucas 1977), used holistic experiences to support the interaction and collaboration of geographically distributed persons. When this fiction comes true, the use of and interaction within AR environments will be as natural as in the real world.

First steps towards a combined vision of Williams and Vinge have already been taken. There has been research on introducing smell (Kim et al. 2011) and touch (Samur 2012) into movie theaters and television. However, one of the most difficult aspects to reproduce is realistic interaction with other (real or virtual) humans. Olson and Olson (2000, 2014) analysed the efforts of geographically distributed teams to use technology to create sense of being in one place and work together. They came to the conclusion that distance matters and that current technology is not mature enough to enable virtual co-presence, and even future technology will struggle to enable this. In their opinion, providing awareness among co-workers and enabling co-reference as well as spatial referencing will remain a challenge. Gaver (1991) stresses the importance of supporting awareness information to help actors shifting from working alone to working together.

Considering current groupware technology, this forecast is still correct. Complex problem solving still requires a team of experts to physically meet and interact with each other. The identification of the problem and the creation of a shared understanding are major challenges for efficiently solving a problem (Piirainen et al. 2012). Unfortunately, due to experts’ availability, critical timing issues or accessibility of a location, it is not always possible to bring a team together to handle a complex situation. While in the novel Rainbows End, such situations are supported with AR technology, current AR experiences are not there yet.

This special issue includes four articles that focus on remote collaboration using different types of AR systems. They advance the state of the art and provide insights about how AR can be used for remote guidance, what are critical design factors, and what is the impact on awareness. Most importantly, they show how close current AR systems are to the future Science Fiction visions described above. After reviewing the state of the art of collaboration using AR systems, this introduction builds upon the articles in this special issue and outlines a research agenda for future work on remote collaboration supported by AR systems.

Background

Several studies have explored the effectiveness of using AR for complex tasks. For individual users, Baird and Barfield (1999) showed that AR can improve the effectiveness of assembly tasks. Similarly, Tang et al. (2003) showed that the use of AR improves performance and reduces mental effort in assembly tasks. Bauer et al. (1999) show that in a distributed setting a remote user can effectively guide and direct a local user’s activities using an AR telepointer. Wang and Dunston (2011) showed that AR systems can improve performance time and mental effort in collaborative design tasks. Dong et al. (2013) found that AR facilitates communication and discussion of engineering processes. Alem and Li (2011) showed that AR can improve the satisfaction with remote collaboration on physical tasks. Recent results on using AR for collaboration among crime scene investigators, indicates that it supports mutual understanding, leads to consensus and supports hypothesis testing (Poelman et al. 2012).

There are several examples of using AR technology for supporting face-to-face collaboration. The Transvision system (Rekimoto 1996) allows multiple users to share computer-generated graphics on a table. The Collaborative Web Space (Billinghurst and Kato 1999, 2002) allows users in the same location to collaboratively browse the web while seeing the real world and communicate naturally about the visited pages. The Studierstube system (Schmalstieg et al. 2002; Szalavári et al. 1998) targets presentations and education and allows users to walk around shared virtual 3D scientific data superimposed over the real world. EMMIE (Butz et al. 1999) uses AR to connect people and devices by visualizing cross-device interactions, search, and privacy status during a meeting. VITA (Benko et al. 2004) supports multiple users in exploring scaled and full-size representations of an archaeological dig. More recent systems considered the use of AR in dynamic emergency response tasks (Nilsson et al. 2009) or within a board game, in which users play social games with physical game pieces using a handheld on a tabletop (Huynh et al. 2009). Similarly, ARVita (Dong et al. 2013) enables multiple users around a table wearing HMDs to observe and interact with dynamic visual simulations of engineering processes. Some of the key lessons learned from these systems are that users can interact with shared AR content as naturally as with physical objects, AR reduces separation between task space and communication space, and that AR enhances natural face-to-face communication cues.

Examples for AR systems that support remote collaboration can be found in Billinghurst and Kato (1999), Höllerer et al. (1999), Stafford et al. (2006), Minatani et al. (2007), Poelman et al. (2012), Datcu et al. (2014a), or Billinghurst and Thomas (2011). For example, WearCom (Billinghurst and Kato 1999) enables users to see remote collaborators as virtual avatars in real space. Höllerer et al. (1999) allow indoor AR users to visualize the locations and paths of outdoor AR users, and create shared annotations. Stafford et al. (2006) introduce a new interaction metaphor called “god-like interaction” which supports multi-scale interaction between outdoor wearable AR users and indoor users using a tabletop projector, and aims at improving sharing of situational and navigational information between remote users. Minatani et al. (2007) describe a system that enables two distributed users to a play a tabletop game in an AR conferencing scenario. Poelman et al. (2012) introduce and evaluate an AR system that allows remote experts to collaborate with local investigators on a crime scene investigation in order to secure evidence. Along the same line Datcu et al. (2014a), present a system that supports situational awareness of cross-organisational teams in the security domain. Some of the key lessons learned from these systems are that AR technology can reproduce some of the spatial cues used in face-to-face collaboration that are normally lost in remote conferencing systems, AR can increase social presence compared to other technologies, and AR also allows remote collaborators to interact naturally in the local user’s real environment (Billinghurst and Thomas 2011).

In addition to the above systems, collaborative AR has also been used in several industry scenarios such as product design, maintenance or factory planning. The ReMote system (Alem et al. 2011; Huang et al. 2013) is a remote assistance system that allows a remote expert to assist and guide in real time a local maintenance operator in a mine site. The remote expert uses their own hands (hand gestures over a live video of the workspace) as well as annotations on still images to explain how the repair /maintenance task needs to be performed. The MagicMeeting system (Regenbrecht et al. 2002) is a multi-user AR system that allows up to four co-located users to have a design zone review meeting. Fata Morgana (Klinker et al. 2002) supports five different scenarios for car designers where they can rotate a car, perform an overview evaluation, focus on details, discuss with colleagues, and compare designs. Roivis (Pentenrieder et al. 2007) is an AR application that supports factory design and planning between co-located users that share a view. DARCC (Hammad et al. 2009) is a distributed AR application that allows multiple users to simulate collaborative construction tasks, e.g., coordinating crane operations. Gauglitz et al. (2014) introduce a tablet-based system that incorporates a touch screen interface through which a remote user can navigate a physical environment and create world-aligned annotations for supporting maintenance. Oda et al. (2013) present a system for remote equipment maintenance. These systems show that collaborative AR technology could have significant impact in design and industrial applications.

In summary, AR technology is becoming mature enough to support a variety of complex collaboration scenarios (Carmigniani et al. 2011; Piekarski and Thomas 2009). However, there are still a number of open issues that need further research. One major issue with regard to AR collaboration is in relation to the presence of remote users (Billinghurst and Thomas 2011). Local users may feel remote controlled by the remote experts, while the experts may feel that they miss something when not physically being present at the scene (Poelman et al. 2012). It is unknown if the proven concepts of awareness support from traditional desktop collaboration situations (Schümmer and Lukosch 2007) can be transferred to AR interaction spaces, or whether completely new approaches are necessary. Billinghurst and Thomas (2011) also point out that further research on how users can interact within collaborative AR systems is necessary. For example, investigating whether bare hand interaction or interaction with a physical object is more effective in AR (Datcu et al. 2015).

Recent research has shown that virtual co-location is in fact possible. Poelman et al. (2012) allows experts to spatially collaborate with others at any other place in the world without traveling and thereby creating the experience of being virtually co-located. In the field of crime scene investigation, a remote expert guides and collaborates with a colleague in a crime scene to collect evidence. In this setting, the remote colleague wears a head-mounted display with a camera and shares the local view with the remote colleague, while both can annotate what they see with virtual objects. Datcu et al. (2014a, b) use AR to establish virtual co-location and improve the telepresence of a remote colleague for police investigations, fire fighting or reconnaissance. In 3DReMoTe (Huang and Alem 2013; Huang et al. 2013), the expert sees a real time a reconstruction & display of a remote workspace via a HMD. The hands of the expert are captured in 3D and co-located in the 3D remote workspace, allowing the expert to assist in spatial tasks. These projects show that AR can enable virtual co-location and allow experts at a distance to interact with local users to perform collaborative tasks.

Content of the special issue

The articles in this special issue present results that continue the line of research as outlined in the previous section. They provide novel insights on remote collaboration in augmented reality and bring us closer towards the combined vision of Tad Williams and Vernor Vinge.

In the first article, Pavel Gurevich, Joel Lanir and Benjamin Cohen describe the design and implementation of TeleAdvisor, a projection-based AR system for remote collaboration. Using a small projector and two cameras mounted on top of a tele-operated robotic arm at the local user’s side, TeleAdvisor allows a remote user to guide the local user in physical tasks that need to be performed. They show that their solution supports flexibility of movements, representational gestures, mixed perspectives, and control of both worker and helper. Based on their findings, they discuss the general implications for design of future AR remote collaboration systems.

Next, Matt Adcock and Chris Gunn present a similar approach for providing remote guidance to a local worker. Their system allows a remote expert to see the workplace from a local worker’s point of view. The remote expert can then draw AR annotations to guide the local worker. The annotations are projected onto the real world and can be made to stick to one physical location. Compared to the previous article, however, Adcock and Gunn do not use a robotic arm, but developed a system that can be worn by the local worker. In their user study, they evaluate the user appreciation of a wearable system compared to a fixed AR system, as well as user appreciation of sticky annotations (e.g., digital annotations that stick to physical objects) compared to digital annotations that move with users movement. Their evaluation suggests that users prefer the sticky annotations, however, they also show that user performance is better when using a fixed setup compared to a wearable setup.

The two previous articles describe solutions to provide remote guidance to a local user. In remote guidance scenarios, the remote user often views the local scene from the point of view of the local user. In the third paper, Matthew Tait and Mark Billinghurst discuss the effect of view independence for the remote user in such a scenario. For that purpose, they present a collaborative AR system that allows a remote user to assist a local user in an object placement task. Unlike most other systems, their AR system allows the remote user to have an independent view into the shared task space. In a user study on remote assistance for a two-dimensional object placement task, they show that increased view independence leads to faster task completion and a decrease in time spent on communication during the task.

The final paper of the special issue by Stephan Lukosch, Heide Lukosch, Dragoş Datcu and Marina Cidota discusses the use of AR for improving situational awareness of teams in the security domain. Lukosch et al. present a collaborative AR system that allows a local and remote user to jointly annotate a local scene using AR. The remote user can provide additional information on the spot. The presented system has been evaluated in two rounds with experts from different operational units in the security domain. The evaluation shows that the AR system can successfully support information exchange in teams operating in the security domain and that it can improve the team situational awareness. The evaluation also showed that the participating experts especially valued the guidance of the remote expert, as well as the possibility to better exchange information among teams from different organization in the security domain.

Research agenda

The four articles in this special issue focus on remote collaboration scenarios and the possibility to provide remote assistance, guidance or additional information to local users. All the articles also highlight the necessity for future research. Gurevich et al. stress that is it necessary to further research the support for remote users and how virtual content is visualized. Adcock and Gunn point out that further research is necessary to understand which tasks are suitable for remote guidance and how the local site is presented to the remote user with regard to issues around presence and efficiency of support. Tait and Billinghurst conclude that further research is needed to improve the usability of the 3D user interface used by the remote expert. Lukosch et al. argue that further research is necessary to understand the impact of AR on team situational awareness. Looking further, they consider user scenarios beyond the remote guidance scenario and ask the question to what extent can AR be used in training situations and how realistic such environments have to be in order to achieve good training outcomes..

Future research on collaboration in AR will thus have to focus on exploring situations that are suitable for virtual co-location. Further experiments need to be conducted to explore how remote users can interact with the local users and how their presence and awareness can be improved. Typically, a remote user perceives the local site via a standard desktop user interface. It is an open question whether more immersive 3D visualisations for remote users by, e.g., using a HMD, will impact the interaction, Presence and awareness in virtual co-location scenarios.

Dubois et al. (1999) discuss the evolution of interaction paradigms from graphical user interfaces to tangible user interfaces, and Augmented Reality. For future collaborative AR systems, there is a need to research which interaction paradigms will be most effective. Starting from proven concepts for 3D user interfaces (Bowman et al. 2004) and remote collaboration (Schümmer and Lukosch 2007), research will have to go beyond current AR systems and create effective 3D user interfaces for groups of multiple local as well as remote users. To enhance the virtual co-location experience and to facilitate the remote guiding task, more research is needed to explore how to further enhance the communication bandwidth between users. Current research efforts around multimodal interactions combining gestures, audio and annotations (Huang and Alem 2013), around using physical objects as interaction means (Datcu et al. 2015), or around how to provide remote users with a full control of their view, where they are able to zoom in and out, change their point of view as they see fit (as pointed out by Tait and Billinghurst in this special issue) are early steps in this direction.

To enhance the perception of presence of remote users, the effect of stimulating other senses (e.g., olfactory) on the perception of presence and situational awareness needs to be researched. First steps in this direction, have been done by (Narumi et al. 2011) who created an AR system that changes the perceived taste of a cookie by overlaying visual and olfactory information onto a real cookie.

Finally, more intuitive interaction possibilities among virtual and real objects need to be developed. Schraffenberger and van der Heide (2013) explored how real objects can affect virtual objects and vice versa. Using several examples, Schraffenberger and van der Heide (2013) argue that virtual and real objects can simulate influences that exist between real entities. Going even further, they give examples on how virtual and real objects can influence each other in new and imaginary ways. Based on these insights, future research needs to explore how distributed users can be empowered to interact with the environment and each other.

In summary, to create an AR system that supports remote collaboration and establishes virtual co-location, research on suitable scenarios, interaction paradigms, Presence and situational awareness needs to be conducted. This special issue on collaboration in Augmented Reality can be seen as a stepping-stone toward the future vision that is outlined in the novels from Williams and Vinge. It is hoped that this special issue will encourage other researchers to become part of the fascinating research and help shape this future.