Skip to main content
SpringerLink
Log in

Mitigation of the Microsoft HoloLens’ hardware limitations for a controlled product assembly process

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Studies have shown that using augmented reality (AR) reduces time and errors by as much as 50% for manufacturing and assembly operations. The use of head-mounted displays (HMDs) is very interesting for these situations as they free up a user’s hands. However, with the technology behind these HMDs still under development, limitations in input, field of view, tracking, and occlusion exist. For AR assembly instructions to be effective, these limitations must be mitigated so that a user receives accurate instructions. To investigate this, an application was developed that guides a user through the assembly of a mock aircraft wing using AR on a Microsoft HoloLens 1. The application displays 3D work instructions, user interfaces, and spatially registered content. To ensure accurate instructions were displayed, techniques were developed to mitigate the HoloLens’ tracking and display limitations. Image tracking was implemented to augment the HoloLens’ limited position determination, stabilize spatially registered virtual content, and account for spatial drift. Also, 3D augmented content was optimized for clarity through the display using specialized computer graphics methods. Lastly, the system provides raw data regarding head position, orientation, and assembly step times. A visualization application was developed that combines this information with wearable sensor data, to examine trends exhibited by a user during the assembly tasks and validate the delivered instructions. The outcomes of this research provided solutions to address limitations of the HoloLens for broad use.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Henderson S, Feiner S (2011) Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Trans Vis Comput Graph 17:1355–1368. https://doi.org/10.1109/TVCG.2010.245

    Article  Google Scholar 

  2. Ockerman JJ, Pritchett AR (1998) Preliminary investigation of wearable computers for task guidance in aircraft inspection. In: IEEE Proceedings of the 2nd International Symposium on Wearable Computers. IEEE Comput. Soc, pp 33–41

  3. Dini G, Mura MD (2015) Application of augmented reality techniques in through-life engineering services. Procedia CIRP 38:14–23. https://doi.org/10.1016/j.procir.2015.07.044

    Article  Google Scholar 

  4. Nee AYC, Ong SK (2013) Virtual and augmented reality applications in manufacturing. IFAC Proc 46:15–26. https://doi.org/10.3182/20130619-3-RU-3018.00637

    Article  Google Scholar 

  5. Hoover M, Miller J, Gilbert S, et al (2018) Evaluating augmented reality assembly instructions delivered via Microsoft HoloLens. In: Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC)

  6. Azuma R (1997) A survey of augmented reality. Presence Teleoperators virtual Environ

  7. Caudell TP, Mizell DW (1992) Augmented reality: an application of heads-up display technology to manual manufacturing processes. In: Proceedings of the Twenty-fifth Hawaii International Conference on System Sciences, pp 659–669 vol 2

  8. Azuma RTA (1997) Survey of augmented reality. Presence Teleoperators Virtual Environ 6:355–385

    Article  Google Scholar 

  9. Van Krevelen DWF, Poelman R (2010) A survey of augmented reality technologies, applications and limitations. …. J Virtual Real 9:1–20. https://doi.org/10.1155/2011/721827

    Article  Google Scholar 

  10. Palmarini R, Erkoyuncu JA, Roy R, Torabmostaedi H (2018) A systematic review of augmented reality applications in maintenance. Robot Comput Integr Manuf 49:215–228. https://doi.org/10.1016/J.RCIM.2017.06.002

    Article  Google Scholar 

  11. Baird KM, Barfield W (1999) Evaluating the effectiveness of augmented reality displays for a manual assembly task. Virtual Reality 4:250–259. https://doi.org/10.1007/BF01421808

    Article  Google Scholar 

  12. Loch F, Quint F, Brishtel I (2016) Comparing video and augmented reality assistance in manual assembly. In: Proceedings - 12th International Conference on Intelligent Environments, IE 2016. pp 147–150

  13. Tatić D, Tešić B (2017) The application of augmented reality technologies for the improvement of occupational safety in an industrial environment. Comput Ind 85:1–10. https://doi.org/10.1016/j.compind.2016.11.004

    Article  Google Scholar 

  14. De Crescenzio F, Fantini M, Persiani F et al (2011) Augmented reality for aircraft maintenance training and operations support. IEEE Comput Graph Appl 31:96–101. https://doi.org/10.1109/MCG.2011.4

    Article  Google Scholar 

  15. Wagner D, Pintaric T, Ledermann F, Schmalstieg D (2005) Towards massively multi-user augmented reality on handheld devices. Springer, Berlin, Heidelberg, pp 208–219

    Google Scholar 

  16. Microsoft (2018) Microsoft HoloLens | The leader in mixed reality technology. https://www.microsoft.com/en-ca/hololens. Accessed 3 Jun 2018

  17. Daqri (2018) Smart Glasses™ - DAQRI. https://daqri.com/products/smart-glasses/. Accessed 3 Jun 2018

  18. Meta (2018) Meta | Augmented Reality. http://www.metavision.com/. Accessed 3 Jun 2018

  19. Leap M Magic Leap. https://www.magicleap.com/. Accessed 6 Jan 2019

  20. Evans G, Miller J, Iglesias Pena M, et al (2017) Evaluating the Microsoft HoloLens through an augmented reality assembly application. In: Degraded environments: sensing, processing, and display. International Society for Optics and Photonics, p 101970V

  21. Tang A, Owen C, Biocca F, Mou W (2003) Comparative effectiveness of augmented reality in object assembly. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp 73–80

  22. Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B (2001) Recent advances in augmented reality. IEEE Comput Graph Appl 21:34–47. https://doi.org/10.1109/38.963459

    Article  Google Scholar 

  23. Comport AI, Marchand E, Pressigout M, Chaumette F (2006) Real-time markerless tracking for augmented reality: the virtual visual servoing framework. In: IEEE Transactions on Visualization and Computer Graphics. pp 615–628

  24. Martin P, Marchand E, Houlier P, Marchal I (2014) Decoupled mapping and localization for augmented reality on a mobile phone. In: Proceedings - IEEE Virtual Reality. IEEE, pp 97–98

  25. Rolland JP, Fuchs H (2000) Optical versus video see-through head-mounted displays in medical visualization. Presence Teleoperators Virtual Environ 9:287–309. https://doi.org/10.1162/105474600566808

    Article  Google Scholar 

  26. Shah MM, Arshad H, Sulaiman R (2012) Occlusion in augmented reality. 8th Int Conf Inf Sci Digit Content Technol (ICIDT 2012) 2:372–378

  27. Wloka MM, Anderson BG (1995) Resolving occlusion in augmented reality. In: Proceedings of the 1995 Symposium on Interactive 3D Graphics - SI3D ‘95. pp 5–12

  28. Microsoft (2018) HoloLens hardware details - Mixed Reality | Microsoft Docs. https://docs.microsoft.com/en-us/windows/mixed-reality/hololens-hardware-details. Accessed 22 May 2018

  29. Unity Technologies (2018) Fast Facts. https://unity3d.com/public-relations. Accessed 10 May 2017

  30. Microsoft (2018) Unity development overview - Mixed Reality | Microsoft Docs. https://docs.microsoft.com/en-us/windows/mixed-reality/unity-development-overview. Accessed 22 May 2018

  31. Soussi L, Spijkerman Z, Jansen S (2016) A case study of the health of an augmented reality software ecosystem: Vuforia. In: International conference of software business. Springer, Cham, pp 145–152

    Chapter  Google Scholar 

  32. Microsoft (2018) Using Vuforia with Unity - Mixed Reality | Microsoft Docs. https://docs.microsoft.com/en-us/windows/mixed-reality/vuforia-development-overview. Accessed 22 May 2018

  33. Rautaray SS, Agrawal A (2012) Vision based hand gesture recognition for human computer interaction: a survey. Artif Intell Rev 43:1–54. https://doi.org/10.1007/s10462-012-9356-9

    Article  Google Scholar 

  34. Cheng H, Yang L, Liu Z (2016) Survey on 3D hand gesture recognition. IEEE Trans Circuits Syst Video Technol 26:1659–1673. https://doi.org/10.1109/TCSVT.2015.2469551

    Article  Google Scholar 

  35. Sekar LPN, Santos A, Beltramello O (2015) IMU drift reduction for augmented reality applications. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). Springer, Cham, pp 188–196

    Google Scholar 

  36. Vuforia (2018) Optimizing target detection and tracking stability. https://library.vuforia.com/articles/Solution/Optimizing-Target-Detection-and-Tracking-Stability.html. Accessed 22 May 2018

  37. MacAllister A, Hoover M, Gilbert S, et al (2017) Comparing visual assembly aids for augmented reality work instructions. In: Comparing Visual Assembly Aids for Augmented Reality Rork Instructions. Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC)

  38. Garbarino M, Lai M, Bender D, et al (2015) Empatica E3 - a wearable wireless multi-sensor device for real-time computerized biofeedback and data acquisition. In: Proceedings of the 2014 4th International conference on wireless mobile communication and healthcare - “Transforming healthcare through innovations in mobile and wireless technologies”, MOBIHEALTH 2014. pp 39–42

  39. Hoover M, MacAllister A, Holub J, et al (2016) Assembly training using commodity physiological sensors. In: Assembly Training using Commodity Physiological Sensors. Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC)

Download references

Funding

This work has been funded by the Boeing Company.

Author information

Authors and Affiliations

  1. Iowa State University, 537 Bissell Road, Ames, IA, 50011, USA

    Jack Miller, Melynda Hoover & Eliot Winer

Authors
  1. Jack Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Melynda Hoover
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eliot Winer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jack Miller.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Miller, J., Hoover, M. & Winer, E. Mitigation of the Microsoft HoloLens’ hardware limitations for a controlled product assembly process. Int J Adv Manuf Technol 109, 1741–1754 (2020). https://doi.org/10.1007/s00170-020-05768-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-020-05768-y

Keywords

Search

Navigation