Teach Me How! Interactive Assembly Instructions Using Demonstration and In-Situ Projection

  • Markus FunkEmail author
  • Lars Lischke
  • Sven Mayer
  • Alireza Sahami Shirazi
  • Albrecht Schmidt
Part of the Cognitive Science and Technology book series (CSAT)


When ordering a product the options to personalize or customize the items have increased over the last years. This flexibility has lead to an increasing number of variants in manufactured products. As storing produced items is expensive, companies tend to produce their products lean, i.e. in smaller lot sizes just when they are needed. This lean manufacturing creates more flexible production environments. In our work, we investigate how human workers can be assisted to work in such demanding environments. Therefore, Augmented Reality systems can be used to provide work instructions. First, in this chapter we provide a comprehensive overview about Augmented Reality approaches to support workers directly at the workplace and introduce an assistive system for providing in-situ instructions. Through three user studies, we evaluate the general impact of in-situ instructions, evaluate three instruction creation strategies, and finally evaluate the created instructions using a real product assembly task.



This work is funded by the German Federal Ministry for Economic Affairs and Energy in the project motionEAP [20], grant no. 01MT12021E. We thank Mathias Hoppe for his work in helping to implement the software and conducting the user study. We further thank Klaus Klein, Michael Spreng, and Johann Hegel from Audi AG.


  1. 1.
    Aleotti J, Caselli S (2006) Robust trajectory learning and approximation for robot programming by demonstration. Robot Auton Syst 54(5):409–413CrossRefGoogle Scholar
  2. 2.
    Antifakos S, Michahelles F, Schiele B (2002) Proactive instructions for furniture assembly. In: UbiComp 2002: ubiquitous computing. Springer, pp 351–360Google Scholar
  3. 3.
    Bannat A, Gast J, Rigoll G, Wallhoff F (2008) Event analysis and interpretation of human activity for augmented reality-based assistant systems. In: 4th international conference on intelligent computer communication and processing, 2008. ICCP 2008. IEEE, pp 1–8Google Scholar
  4. 4.
    Barna J, NovakovaMarcincinova L, Novak-Marcincin J, Fecova V, Janak M, Torok J (2012) Open source tools in assembling process enriched with elements of augmented reality. In: Proceedings of the 2012 virtual reality international conference. ACM, p 2Google Scholar
  5. 5.
    Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. In: Computer vision-ECCV 2006. Springer, pp 404–417Google Scholar
  6. 6.
    Billard A, Calinon S, Dillmann R, Schaal S (2008) Robot programming by demonstration. In: Springer handbook of robotics. Springer, pp 1371–1394Google Scholar
  7. 7.
    Blanke U, Schiele B, Kreil M, Lukowicz P, Sick B, Gruber T (2010) All for one or one for all? Combining heterogeneous features for activity spotting. In: 2010 8th IEEE international conference on Pervasive computing and communications workshops (PERCOM Workshops). IEEE, pp 18–24Google Scholar
  8. 8.
    Büttner S, Sand O, Röcker C (2015) Extending the design space in industrial manufacturing through mobile projection. In: Proceedings of the 17th international conference on human-computer interaction with mobile devices and services adjunct. ACM, pp 1130–1133Google Scholar
  9. 9.
    Büttner S, Mucha H, Funk M, Kosch T, Aehnelt M, Robert S, Röcker C (2017) The design space of augmented and virtual reality applications for assistive environments in manufacturing: a visual approach. In: Proceedings of the 10th international conference on pervasive technologies related to assistive environments. ACM, pp 433–440Google Scholar
  10. 10.
    Büttner S, Funk M, Sand P, Röcker C (2016) Using head-mounted displays and in-situ projection for assistive systems: a comparison. In: Proceedings of the 9th ACM international conference on pervasive technologies related to assistive environments. ACM, p 44Google Scholar
  11. 11.
    Caudell TP, Mizell DW (1992) Augmented reality: an application of heads-up display technology to manual manufacturing processes. In: Proceedings of the twenty-fifth hawaii international conference on system sciences, vol 2. IEEE, pp 659–669Google Scholar
  12. 12.
    Collett T, MacDonald BA (2006) Developer oriented visualisation of a robot program. In: Proceedings of the 1st ACM SIGCHI/SIGART conference on human-robot interaction. ACM, pp 49–56Google Scholar
  13. 13.
    Fiorentino M, de Amicis R, Monno G, Stork A (2002) Spacedesign: a mixed reality workspace for aesthetic industrial design. In: Proceedings of the 1st international symposium on mixed and augmented reality. IEEE Computer Society, p 86Google Scholar
  14. 14.
    Funk M, Korn O, Schmidt A (2015) Enabling end users to program for smart environments. In: Proceedings of the CHI 2015—workshop on end user development in the internet of things era 12.2, pp 9–14Google Scholar
  15. 15.
    Funk M, Kosch T, Schmidt A (2016) Interactive worker assistance: comparing the effects of in-situ projection, head-mounted displays, tablet, and paper instructions. In: Proceedings of the 2016 ACM international joint conference on pervasive and ubiquitous computing. ACM, pp 934–939Google Scholar
  16. 16.
    Funk M, Mayer S, Schmidt A (2015) Using in-situ projection to support cognitively impaired workers at the workplace. In: Proceedings of the 17th international ACM SIGACCESS conference on computers and accessibilityGoogle Scholar
  17. 17.
    Funk M, Schmidt A (2015) Cognitive assistance in the workplace. Pervasive Comput IEEE 14(3):53–55CrossRefGoogle Scholar
  18. 18.
    Funk M, Kosch T, Greenwald SW, Schmidt A (2015) A benchmark for interactive augmented reality instructions for assembly tasks. In: Proceedings of the 14th international conference on mobile and ubiquitous multimedia. ACM, pp 253–257Google Scholar
  19. 19.
    Funk M, Bächler A, Bächler L, Korn O, Krieger C, Heidenreich T, Schmidt A (2015) Comparing projected in-situ feedback at the manual assembly workplace with impaired workers. In: Proceedings of the 8th ACM international conference on pervasive technologies related to assistive environments. ACM, p 1Google Scholar
  20. 20.
    Funk M, Kosch T, Kettner R, Korn O, Schmidt A (2016) Motioneap: an overview of 4 years of combining industrial assembly with augmented reality for industry 4.0. In: Proceedings of the 16th international conference on knowledge technologies and datadriven businessGoogle Scholar
  21. 21.
    Funk M, Shirazi AS, Mayer S, Lischke L, Schmidt A (2015) Pick from here!: an interactive mobile cart using in-situ projection for order picking. In: Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing. ACM, pp 601–609Google Scholar
  22. 22.
    Guo A, Raghu S, Xie X, Ismail S, Luo X, Simoneau J, Gilliland S, Baumann H, Southern C, Starner T (2014) A comparison of order picking assisted by head-up display (HUD), cart-mounted display (CMD), light, and paper pick list. In: Proceedings of the 2014 ACM international symposium on wearable computers. ACM, pp 71–78Google Scholar
  23. 23.
    Hahn J, Ludwig B, Wolff C (2015) Augmented reality-based training of the PCB assembly process. In: Proceedings of the 14th international conference on mobile and ubiquitous multimedia. ACM, pp 395–399Google Scholar
  24. 24.
    Hardy J, Alexander J (2012) Toolkit support for interactive projected displays. In: Proceedings of the 11th international conference on mobile and ubiquitous multimedia. ACM, p 42Google Scholar
  25. 25.
    Hart SG, Staveland LE (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv Psychol 52:139–183CrossRefGoogle Scholar
  26. 26.
    Henderson S, Feiner S (2011) Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Trans Visual Comput Graph 17(10):1355–1368CrossRefGoogle Scholar
  27. 27.
    Hermann M, Pentek T, Otto B. Design principles for industrie 4.0 scenarios: a literature reviewGoogle Scholar
  28. 28.
    Klompmaker F, Nebe K, Fast A (2012) dSensingNI: a framework for advanced tangible interaction using a depth camera. In: Proceedings of the sixth international conference on tangible, embedded and embodied interaction. ACM, pp 217–224Google Scholar
  29. 29.
    Korn O, Schmidt A, Hörz T (2013) Augmented manufacturing: a study with impaired persons on assistive systems using in-situ projection. In: Proceedings of the 6th international conference on pervasive technologies related to assistive environments. ACM, p 21Google Scholar
  30. 30.
    Korn O, Funk M, Abele S, Hörz T, Schmidt A (2014) Context-aware assistive systems at the workplace: analyzing the effects of projection and gamification. In: Proceedings of the 7th international conference on pervasive technologies related to assistive environments. ACM, p 8Google Scholar
  31. 31.
    Kubitza T, Schmidt A (2015) Towards a toolkit for the rapid creation of smart environments. In: End-user development. Springer, pp 230–235Google Scholar
  32. 32.
    Lee C-H, Bonnani L, Selker T (2005) Augmented reality kitchen: enhancing human sensibility in domestic life. In: ACM SIGGRAPH 2005 posters. ACM, p 60Google Scholar
  33. 33.
    Lieberman H (2001) Your wish is my command: programming by example. Morgan KaufmannGoogle Scholar
  34. 34.
    Linder N, Maes P (2010) LuminAR: portable robotic augmented reality interface design and prototype. In: Adjunct proceedings of the 23nd annual ACM symposium on user interface software and technology. ACM, pp 395–396Google Scholar
  35. 35.
    Lucke D, Constantinescu C, Westkämper E (2008) Smart factory-a step towards the next generation of manufacturing. In: Manufacturing systems and technologies for the new frontier. Springer, pp 115–118Google Scholar
  36. 36.
    Marinos D, Wöldecke B, Geiger C (2013) Prototyping natural interactions in virtual studio environments by demonstration: combining spatial mapping with gesture following. In: Proceedings of the virtual reality international conference: laval virtual. ACM, p 2Google Scholar
  37. 37.
    Myers BA (1986) Creating dynamic interaction techniques by demonstration. In: ACM SIGCHI bulletin 17.SI, pp 271–278 (1986)Google Scholar
  38. 38.
    Pinhanez C (2001) The everywhere displays projector: a device to create ubiquitous graphical interfaces. In: Ubicomp 2001: ubiquitous computing. Springer, pp 315–331Google Scholar
  39. 39.
    Rüther S, Hermann T, Mracek M, Kopp S, Steil J (2013) An assistance system for guiding workers in central sterilization supply departments. In: Proceedings of the 6th international conference on pervasive technologies related to assistive environments. ACM, p 3Google Scholar
  40. 40.
    Salonen T, Sääski J, Hakkarainen M, Kannetis T, Perakakis M, Siltanen S, Potamianos A, Korkalo O, Woodward C (2007) Demonstration of assembly work using augmented reality. In: Proceedings of the 6th ACM international conference on image and video retrieval. ACM, pp 120–123Google Scholar
  41. 41.
    Schmidt A (2000) Implicit human computer interaction through context. Pers Technol 4(2–3):191–199Google Scholar
  42. 42.
    Tang A, Owen C, Biocca F, Mou W (2003) Comparative effectiveness of augmented reality in object assembly. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 73–80Google Scholar
  43. 43.
    Ward JA, Lukowicz P, Troster G, Starner TE (2006) Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE Trans Pattern Anal Mach Intell 28(10):1553–1567CrossRefGoogle Scholar
  44. 44.
    Wilson AD (2004) TouchLight: an imaging touch screen and display for gesture-based interaction. In: Proceedings of the 6th international conference on Multimodal interfaces. ACM, pp 69–76Google Scholar
  45. 45.
    Wilson AD (2010) Using a depth camera as a touch sensor. In: ACM international conference on interactive tabletops and surfaces. ACM, pp 69–72Google Scholar
  46. 46.
    Wohlgemuth W, Triebfürst G (2000) ARVIKA: augmented reality for development, production and service. In: Proceedings of DARE 2000 on designing augmented reality environments. ACM, pp 151–152Google Scholar
  47. 47.
    Zauner J, Haller M, Brandl A, Hartman W (2003) Authoring of a mixed reality assembly instructor for hierarchical structures. In: The second IEEE and ACM international symposium on mixed and augmented reality, 2003. Proceedings. IEEE, pp 237–246Google Scholar
  48. 48.
    Zollner R, Rogalla O, Dillmann R, Zollner M (2002) Understanding users intention: programming fine manipulation tasks by demonstration. In: IEEE/RSJ international conference on intelligent robots and systems, 2002, vol 2. IEEE, pp 1114–1119Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  • Markus Funk
    • 1
    Email author
  • Lars Lischke
    • 1
  • Sven Mayer
    • 1
  • Alireza Sahami Shirazi
    • 2
  • Albrecht Schmidt
    • 1
  1. 1.University of StuttgartStuttgartGermany
  2. 2.Yahoo Inc.SunnyvaleUSA

Personalised recommendations