Advertisement

Mixed Reality in Action - Exploring Applications for Professional Practice

  • Adam NowakEmail author
  • Mikołaj Woźniak
  • Michał Pieprzowski
  • Andrzej Romanowski
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 941)

Abstract

Mixed reality technologies has been emerging rapidly in the recent years, not only in terms of improving the equipment and software, but also in employing the current state into multiple practical applications. One of the most successful endeavours of introducing a commercial product for mixed reality is the HoloLens by Microsoft. In this paper, we describe the technology and discuss multiple practical applications among various branches of medicine, science and industry. We present the results of our desk research among various ways to employ the mixed reality into practice and discuss the new openings that arise along the development of the technology. The openings described consist of various interaction methods and applications for CSCW.

1 Introduction

Visualisation and display technologies are one of the most emerging branches in computer science nowadays. Among multiple techniques of displaying graphical representations, the virtual reality has drawn huge attention of the researchers in last decade. There is still much room for exploration in the domain of virtual displays, which pose new possibilities and prospects for application towards various tasks, especially where stationary screen-based display is considered undesired. New opportunities may be investigated both in terms of increased immersion, enhanced mobility and three dimensional perception.

Traditional VR solutions, such as Oculus Rift, HTC Vive, and Samsung Gear were developed with the focus set mainly on entertainment solutions. Manufacturers struggled to offer highly immersive user experiences, which will enable the users to dive into the presented action. [8] The branch of gaming industry which develop VR-based software is rapidly growing in recent years and provides more and more engaging digital products. However, the constraints of the technology concerning limited context-awareness of the user while participating in the experience poses additional need for external supervision and providing safe surroundings. Combined with relatively high price of the device, the technology has not yet established its position as an everyday companion for domestic users. On the other hand, the solution became very popular for applications in exhibition displays and boosted the development of various branches of digital art.

The advantages of head-based, highly immersive display are numerous, while the environmental constraints highly affect the usability, which draw users away from applying it for alleviating everyday problems. [24] The potential of the technology for professional purposes is equally promising, opening brand new opportunities for extensive data analysis, new ways of medical therapy and many more. One of the most successful endeavours towards overcoming those constraints while maintaining the key functionality was introducing HoloLens by Microsoft. Turning towards mixed-reality approach, which augment the holographic projection onto the vision perceived by user’s sight. Therefore, the comfort of on-head, adaptive display is maintained while not excluding full context awareness. In this paper, we discuss in detail the construction, user experience design and possible applications of Microsoft HoloLens in various branches of science and industry.

2 Hardware Details

2.1 Virtual Reality vs Augmented Reality

Virtual reality and Augmented reality are similar technologies focused on displaying artificial objects to the user. Virtual reality devices are based on headsets fulfilling whole field of sight, disabling the awareness of any outer factors. When mounted, the user is entering purely virtual environment created with digital objects and landscapes. Complete separation of sense of sight and perceived body movements often causes virtual reality sickness, symptoms of which (discomfort, headache, stomach awareness, nausea) [2, 25] are similar to motion sickness [14] (Fig. 1).
Fig. 1.

Scale of enhanced reality [16].

The interaction with the projected artefacts is performed through rotation of users’ head and actions performed with in-hand controllers. The hand controllers are equipped with gyroscopic sensors and tilt sensors, which translate the movements into virtual world.

Multiple VR solutions bare its users with cable connection to the unit which operates the software, which highly limits the mobility and freedom of movement [11]. This issue has been faced applying mobile devices as the units for VR, yet such solutions suffer from lower performance due to limited computational resources and power issues [6].

Augmented reality technology present different scope, connecting virtual objects with real-world environment, overlaying artificial artefacts on real surfaces and objects. Holographic display provided by the AR device enable the user to experience the visualisation while remaining fully context-aware. The approach of presenting a full holographic display, not limited to few artefacts, is also referred as the mixed reality. The mixed reality devices are capable of real-time scanning and interpreting the users’ surroundings, such as recognition of walls, floors, furniture and other obstacles. Therefore, the projections can be displayed with regard to the real environment, simulating the real-world objects. The dynamic analysis enables distinct context interpretation, enabling the user to remain 3D relations between real and artificial objects.

Navigation within the projection is based on interpreting the hand gestures using image recognition, which employ unique movements, such as air tap or bloom, for controlling the device interface (Figs. 2 and 3).
Fig. 2.

Air tap gesture

Fig. 3.

Bloom gesture

The HoloLens device, being one of the most successful commercial mixed reality devices act as an independent device, with the in-built computational unit. The device performs voice recognition and is able to provide audible interactions. Multiple devices are able to cooperate using server-based sharing facilities. The data change on one device is sent to the server and broadcasted to all the devices in a session [12] (Fig. 4).
Fig. 4.

User working with HoloLens device

3 User Experience of Mixed Reality

The ultimate goal for designers approaching developing mixed-reality systems is to provide an engaging and immersive experience, which will enable the user to properly approach the goals via the visual interface with no peripherals needed. The main challenge is to provide an intuitive manner of interaction, which will employ well-known gestural patterns for interacting with the display.

HoloLens introduces brand new manner of interaction with technology, forcing the users to step out of their comfort zone and establish new habits. Therefore it is crucial to properly cover the user experience design, so the device is not just a tech-savvy gadgetry, but would offer numerous opportunities and will to be employed as an everyday companion.

3.1 Pursuing an Engaging Experience

According to Nielsen’s three dimensional conceptual model of user experience [17], the HoloLens device has the emphasis put on developing the engagement layer. The interface is design to enhance the feeling of being augmented with a holographic display in your eyes. The gestural manner of interaction creates immersive perception, as it employs the natural, physical action for controlling the visually perceived surrounding. Using the device is engaging, as the narrative is constructed in an appealing way, making user feel as a participant in a futuristic experience. The system is highly adaptive, as it employs the users surrounding as a part of displaying technique.

Unlike in traditional VR approach, alongside high immersion, user does not get isolated form the world around him. Therefore, task combining operation on physical objects with a support of holographic display becomes possible. Moreover, the interface enables realising collaborative tasks with other people, non-equipped with a similar device. That opens new possibilities for computer supported teamwork in broad range of applications, especially where hand-operated displays are not sufficiently convenient.

4 Applications of Mixed Reality

Nevertheless the mixed reality is still considered among gadgetry, the innovative approach towards displaying and convenient manner of operation makes it suitable for numerous applications. New ways of utilising the technology are being explored, with substantial seen among the industrial and medical branches of both science and practice. Having its origins in VR setups, the device may also perform as a home entertainment tool or be applied in various types of digital art. In this paper we present a choice of most promising and innovative endeavours towards applying HoloLens and related technologies into everyday use.

4.1 Medical Use of Mixed Reality

Numerous applications were proposed and investigated for applying extended reality into medical practise. The HoloLens device may be helpful both for the doctors as a supportive tool for diagnostic procedures, as well as directly aid the therapy processes.

The holographic display brings brand new openings in terms of data visualisation and analysis. Artanim Foundation proposed an augmented reality visualisation of joint movements for rehabilitation and sports medicine [4]. Dabarba et al. describe ways to depict joint movements using AR technology. The system is based on optical motion capture and reconstruction of body structures. Bones are presented as real-time simulated holograms, presented similarly to X-ray vision. Gathered information is saved and can be further processed. The system provides an aid to rehabilitation process and is useful for assessing patient’s progress. The system may also help in preventing injuries by analysing trajectories of fast movements among athletes.

Another branch of medicine that can benefit form active use of holographic displays is the anatomic pathology. Hanna et al. [10] shows possibilities among MR applications for medicine. The device was tested by pathologists during autopsy. All the time the access to diagrams, annotations, and voice instruction was given. 3D-scanned gross pathology specimens can be easily manipulated thanks to holographic view. The HoloLens helped in locating important pathological findings. Authors ensure that the device was comfortable to wear and easy to use with sufficient computing power and high-resolution imaging.

Preliminary use of HoloLens glasses in surgery of liver cancer is another promising opening for employing the device into medical practice [23]. Three dimensional model of liver cancer was created during operation by using magnetic resonance data and combining it with VR technology. Moreover, surgical planning was achieved and matched with the target organ during operation. 3D models are likely to act as an informational aid for appropriate planning of the surgery.

An interesting endeavour were made towards X-ray free endovascular interventions - using mixed reality for on-line holographic visualisation [13]. Endovascular interventions require catheter tracking, which is usually achieved by X-ray. Yet, the radiation generated during the process is considered harmful and efforts are paid to avoid this approach. A 3D holographic view of the vascular system is presented as an alternative. Patient’s surface and vascular tree is extracted before surgery using computer tomography and registered by magnetic tracking system. The system was evaluated on a phantom and achieved promising results. While there is much room for improvement, 3D-view displays are likely to substitute the radiation-based techniques in the future.

Mixed reality has also been applied for therapy of Alzheimer’s disease [1]. A joint project with a team of neurologists resulted in creating a set of activities for training short term and spatial memory using HoloLens. Presented exercises were similar to popular “Memory Game”, consisting finding two identical objects hidden in boxes. With growth of difficulty level, boxes were placed chaotically.

Mixed reality device can also be applied to monitor pulse and other bodily functions [15]. Authors present a system for measuring and visualisation of blood flow and vital signs. The system works in real-time and does not require any contact sensors. Recovering pulse signals is possible thanks to the combination of a webcam, imaging ballistocardiography, and remote imaging photoplethysmography. Users can observe their own heart rate as well as the heart rate of other people.

Another disease that might employ MR in its therapy is the amblyopia, so-called the lazy eye syndrome [18]. An interactive game with alternating display, accordingly to the eye being treated, was constructed as a form of exercise, alternative to the classic, eyepatch based approach. The application displays more active visuals to the “lazy eye”, while leaving the more static objects to the other one. Therefore, an increased activity of the amblyopic eye is obtained, while maintaining the binocular vision throughout the exercise. The system was introduced, but is lacking a clinical proof based on patient studies.

4.2 Industrial and Engineering Applications

Holographic projection can play a huge role in control and analysis of industrial processes. The real-time visualisation approach opens new possibilities for advanced control, and the simulation systems can be used for both testing new approaches, as well as educating the new breed of professionals.

An attempt towards improving engineering education using augmented reality environment has been made by Guo [9]. The HoloLens device has been used as a tool to improve students’ results with understanding manual material handling (MMH). Experiment was conducted with the control group, while the experimental group’s students had the opportunity to strengthen their understanding with use of AR application developed by the authors. The AR group performed better, the mean score of traditional class was 62/100, whilst AR group 76.25/100. The results suggest that AR software might be advantageous as a tool for classroom use.

Investigations of head-mounted displays for industrial practice were investigated by Dhiman et al. [5]. The aim of the research was to take closer look at advantages and disadvantages of head-mounted display (HDM) comparing to projection-based display on assembly assistance system [26]. Authors discuss problems of limited field of view, weight of the device, design of the workplace, distance for hologram placement. Noticed positives consist of mobility of the workplace, extended integration modalities, possibility to display content in arbitrary 3D space.

Another branch of industry in which the potential of on-head displays is considered beneficial is the industrial process control. To enable precise and quick insights, techniques such as electrical capacitance tomography are used. In order to enable efficient analysis, it is required to rapidly visualise the collected measurement data in a clear and interactive way. Since the analysis is often performed on-site, which requires high context-awareness and a good blend of virtual display based analysis and physical actions performed on the equipment under investigation [21]. Additional opportunities open when concerning crowd sourced analysis of multi-layer tomographic data [22].

4.3 Alternative Applications

A novel approach was proposed for enhancing cultural tourism by a mixed reality application for outdoor navigation and information browsing using immersive devices. Debandi et al. [3] introduces a mixed reality application providing information on a city scale. The main purpose is presenting and describing monuments, buildings and artworks. In consequence, cultural tourism may be enhanced. Framed images and videos are processed by a remote application. In this way, known objects can be identified and presented to the user. Defined gestures allow choosing the subject and seeing augmented contents such as video or text audio. Objects supported by the mixed reality application can be presented with three dimensional contents and combined with the real world.

Another cultural activity was proposed by Pollalis et al. [20] Virtual archaeological artefacts are displayed to the user and facilitate learning and engagement with exhibition. User can feel more like an archaeologist not only by watching but also exploring artefacts. Commonly insurmountable interactions with artefacts, including picking up, scaling, rotating, moving, are now available with digital copies of real ancient objects. Detailed information might be requested for chosen artefact and presented as text and voice. Through interactive exhibition, the application advocates tourists and visitors along with sparking their imagination. The application connects virtual objects with the original context of the exhibition.

One of the most sophisticated areas of applications seems to be military. Article presents approach in Italian Air Force tending to use mixed reality to the Command Control Communications Computers and Intelligence (C4I) systems [19]. MR may be used by technical staff for training, maintenance and simulation purposes. Introducing virtual ones may significantly reduce the costs and induct more flexibility. Moreover, rationalisation of maintenance organisation can be performed. The device was tested in different scenarios in environmental conditions with provided simulation of interventions. The results are very promising, because the planned maintenance activities were accomplished.

5 Perspectives and New Tendencies

Table 1.

Juxtaposition of applications with criteria

Gesture interaction

Cooperative work

Spatial understanding

Voice

Additional hardware

Visualisation of joint movements [4]

+/−

+

Anatomic pathology [10]

+

+/−

Surgery of liver cancer [23]

+

X-ray free endovascular interventions [13]

+

+

+/-

Therapy of Alzheimer’s disease [1]

+

+

+

Measurement of pulse and vital functions [15]

+/−

Lazy eye syndrome [18]

Improving engineering education [9]

+

Industrial practice [5]

+

+/−

Cultural tourism [3]

+

+

+

Virtual archaeological artefacts [20]

+

Italian Air Force [19]

+

+

The most advantageous asset of mixed reality technology is the technology of holographic display, which poses multiple opportunities towards embedding displayed artefacts toward real environments. Furthermore, it is the gestural interaction which makes the head-mounted MR solutions significantly more convenient to operate than common VR solutions. However, at current stage the range of gestures implemented is vastly limited (Table 1).

Despite the technological limitations, the possibility to merge virtual and real environments opens brand new opportunities in various branches of industry and science. Computer-supported cooperative work is another field that can highly benefit from those new interaction models. However, applications requiring multiple HoloLens devices may be considered barely affordable.

Another branch of applications which may significantly develop through appying MR is the spatial analysis of objects and processes. Through understanding of the environment the application might be more authentic and impressive, although it is not extensively employed.

Voice control is another way of controlling and interacting with mixed reality applications.

The gestural interactions could possibly benefit from employing EMG-driven peripherals. Therefore, wider range of gestures could be easily introduced to the interface. Application of such devices was already tested in terms of application for display-free text edition [7] and audio control [28]. These are also the gaze-tracking systems that could highly benefit from employing MR devices into common practice [27].

6 Discussion and Summary

The mixed reality technology has been truly emerging in recent years, which is proved by developing multiple applications of the device in various branches of industry, science and medicine. The solutions applied still require further development and introducing new methods for increased efficiency. More user studies are advisable, so to make system not only fulfil the tasks, but also become more convenient to its users.

Current studies show that mixed reality broadens the horizon of new therapeutical methods accessible, in various branches of medical treatment. It can also bring the therapy to patient’s home, due to mobility and user-friendly design. Further research is conducted toward establishing the most effective way to employ VR and Mixed Reality technologies to treatment proceedings, since broad sample studies are required to fully confirm the effects of this alternative approach.

Furthermore, the industry can highly benefit from deeper integration of MR technologies. The on-head displays can revolutionise the area of dynamic process control and rapid data visualisation. Control systems implemented onto holographic display can be especially effective, when being on-site is desired and the additional devices will pose additional challenge for the professional to handle.

References

  1. 1.
    Aruanno, B., Garzotto, F., Rodriguez, M.C.: HoloLens-based mixed reality experiences for subjects with Alzheimer’s disease. In: Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter, CHItaly 2017, pp. 15:1–15:9. ACM, New York (2017)Google Scholar
  2. 2.
    Brooks, J.O., Goodenough, R.R., Crisler, M.C., Klein, N.D., Alley, R.L., Koon, B.L., Logan, W.C., Ogle, J.H., Tyrrell, R.A., Wills, R.F.: Simulator sickness during driving simulation studies. Accid. Anal. Prev. 42(3), 788–796 (2010). Assessing Safety with Driving SimulatorsCrossRefGoogle Scholar
  3. 3.
    Debandi, F., Iacoviello, R., Messina, A., Montagnuolo, M., Manuri, F., Sanna, A., Zappia, D.: Enhancing cultural tourism by a mixed reality application for outdoor navigation and information browsing using immersive devices. In: IOP Conference Series: Materials Science and Engineering, vol. 364, p. 012048, June 2018Google Scholar
  4. 4.
    Debarba, H.G., de Oliveira, M.E., Ladermann, A., Chague, S., Charbonnier, C.: Augmented reality visualisation of joint movements for physical examination and rehabilitation. In: Proceeding of 2018 IEEE Conference on Virtual Reality and 3D User Interfaces. IEEE (2018)Google Scholar
  5. 5.
    Dhiman, H., Martinez, S., Paelke, V., Röcker, C.: Head-mounted displays in industrial AR-applications: ready for prime time? In: Nah, F.F.-H., Xiao, B.S. (eds.) HCI in Business, Government, and Organizations, pp. 67–78. Springer, Cham (2018)CrossRefGoogle Scholar
  6. 6.
    Pena, M.I., MacAllister, A., Winer, E., Evans, G., Miller, J.: Evaluating the Microsoft HoloLens through an augmented reality assembly application (2017)Google Scholar
  7. 7.
    Ghosh, D., Foong, P.S., Zhao, S., Chen, D., Fjeld, M.: EDITalk: towards designing eyes-free interactions for mobile word processing. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018, pp. 403:1–403:10. ACM, New York (2018)Google Scholar
  8. 8.
    Grigore, P.C., Burdea, C.: Virtual Reality Technology (2003)Google Scholar
  9. 9.
    Guo, W.: Improving engineering education using augmented reality environment. In: Zaphiris, P., Ioannou, A. (eds.) Learning and Collaboration Technologies. Design, Development and Technological Innovation, pp. 233–242. Springer, Cham (2018)CrossRefGoogle Scholar
  10. 10.
    Hanna, M., Ahmed, I., Nine, J., Prajapati, S., Pantanowitz, L.: Augmented reality technology using Microsoft HoloLens in anatomic pathology. Arch. Pathol. Lab. Med. 142, 638–644 (2018)CrossRefGoogle Scholar
  11. 11.
    Kolasinski, E.: Technical report 1027: simulator sickness in virtual environments (1995)Google Scholar
  12. 12.
    Kress, B.C., Cummings, W.J.: 11–1: invited paper: towards the ultimate mixed reality experience: HoloLens display architecture choices. SID Symp. Dig. Tech. Pap. 48(1), 127–131 (2017)CrossRefGoogle Scholar
  13. 13.
    Kuhlemann, I., Kleemann, M., Jauer, P., Schweikard, A., Ernst, F.: Towards X-ray free endovascular interventions - using HoloLens for on-line holographic visualisation. Healthc. Technol. Lett. 4, 184–187 (2017)CrossRefGoogle Scholar
  14. 14.
    LaViola Jr., J.J.: A discussion of cybersickness in virtual environments. SIGCHI Bull. 32(1), 47–56 (2000)CrossRefGoogle Scholar
  15. 15.
    McDuff, D.J., Hurter, C., González-Franco, M.: Pulse and vital sign measurement in mixed reality using a HoloLens. In: VRST (2017)Google Scholar
  16. 16.
    Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Trans. Inf. Syst. E77-D(12), 1321–1329 (1994)Google Scholar
  17. 17.
    Nielsen, J. (ed.): Coordinating User Interfaces for Consistency. Morgan Kaufmann, San Francisco (1989)zbMATHGoogle Scholar
  18. 18.
    Nowak, A., Woźniak, M., Pieprzowski, M., Romanowski, A.: Towards amblyopia therapy using mixed reality technology. In: Proceedings of the Federated Conference on Computer Science and Information Systems, pp. 279–282 (2018)Google Scholar
  19. 19.
    Piedimonte, P., Ullo, S.L.: Applicability of the mixed reality to maintenance and training processes of C4I systems in Italian Air Force. In: 2018 5th IEEE International Workshop on Metrology for AeroSpace (MetroAeroSpace), pp. 559–564, June 2018Google Scholar
  20. 20.
    Pollalis, C., Fahnbulleh, W., Tynes, J., Shaer, O.: HoloMuse: enhancing engagement with archaeological artifacts through gesture-based interaction with holograms. In: Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction, TEI 2017, pp. 565–570. ACM, New York (2017)Google Scholar
  21. 21.
    Romanowski, A.: Contextual processing of electrical capacitance tomography measurement data for temporal modeling of pneumatic conveying process. In: Proceedings of the 2018 Federated Conference on Computer Science and Information Systems, FedCSIS 2018. ACSIS. IEEE (2018, in press)Google Scholar
  22. 22.
    Romanowski, A., Grudzien, K., Chaniecki, Z., Wozniak, P.: Contextual processing of ECT measurement information towards detection of process emergency states. In: 2013 13th International Conference on Hybrid Intelligent Systems (HIS), pp. 291–297 (2013)Google Scholar
  23. 23.
    Shi, L., Luo, T., Zhang, L., Kang, Z., Chen, J., Wu, F., Luo, J.: Preliminary use of HoloLens glasses in surgery of liver cancer. Zhong nan da xue xue bao. Yi xue ban = J. Cent. South Univ. Med. Sci. 43, 500–504 (2018)Google Scholar
  24. 24.
    Shneiderman, B., Plaisant, C.: Designing the User Interface, 4th edn. Pearson Addison Wesley, Boston (2010)Google Scholar
  25. 25.
    Stanney, K.M., Kennedy, R.S., Drexler, J.M.: Cybersickness is not simulator sickness. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 41(2), 1138–1142 (1997)CrossRefGoogle Scholar
  26. 26.
    Wang, Z.B., Ong, S.K., Nee, A.Y.C.: Augmented reality aided interactive manual assembly design. Int. J. Adv. Manuf. Technol. 69(5), 1311–1321 (2013)CrossRefGoogle Scholar
  27. 27.
    Wojciechowski, A., Fornalczyk, K.: Single web camera robust interactive eye-gaze tracking method. Bull. Pol. Acad. Sci. Tech. Sci. 63(4), 879–886 (2015)Google Scholar
  28. 28.
    Woźniak, M., Polak-Sopińska, A., Romanowski, A., Grudzień, K., Chaniecki, Z., Kowalska, A., Wróbel-Lachowska, M.: Beyond imaging - interactive tabletop system for tomographic data visualization and analysis. In: Karwowski, W., Trzcielinski, S., Mrugalska, B., Di Nicolantonio, M., Rossi, E. (eds.) Advances in Manufacturing, Production Management and Process Control, pp. 90–100. Springer, Cham (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Adam Nowak
    • 1
    Email author
  • Mikołaj Woźniak
    • 1
  • Michał Pieprzowski
    • 1
  • Andrzej Romanowski
    • 1
  1. 1.Institute of Applied Computer ScienceLodz University of TechnologyLodzPoland

Personalised recommendations