Haptic Augmented Reality (HapticAR) for assembly guidance

  • J. C. ArbeláezEmail author
  • Roberto Viganò
  • Gilberto Osorio-Gómez
Original Paper


The use of Augmented Reality (AR) to support assembly tasks has been an area of interest from its origins in the 90s. Since then, the benefits that this technology could bring to assembly-related tasks have been shown. And, although several advances have been done in different areas such as software, hardware, and human interaction, there are still some problems that have not allowed AR to expand and reach its full potential. Thereby, authors propose a real-time vibrotactile guidance method based on the Gestalt continuity principle and developed a Haptic Augmented Reality application with a low-cost configuration to evaluate the support of the proposed method in assembly tasks. Thus, it potentially overcomes the existing visual issues of AR allowing the user to focus on the task and avoid over-reliance into the technology. The proposed system recognizes the different parts and sub-assemblies, generates the instructions to perform the assembly based on the target position and rotation of each part and verifies the assembly. Additionally, a test was conducted to guide the user in positioning a part, obtaining a high accuracy of rotation and location placement. Also, different functions of the application were tested and the results are suitable for supporting the user guidance.


Augmented Reality Haptic Assembly guidance User interaction Object recognition 



  1. 1.
    Ahmaniemi, T.T., Lantz, V.T.: Augmented reality target finding based on tactile cues. In: Proceedings of the 2009 International Conference on Multimodal Interfaces, pp. 335–342. ACM (2009)Google Scholar
  2. 2.
    Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.: Recent advances in augmented reality. IEEE Comput. Graph. Appl. 21(6), 34–47 (2001)Google Scholar
  3. 3.
    Baird, K.M., Barfield, W.: Evaluating the effectiveness of augmented reality displays for a manual assembly task. Virtual Real. 4(4), 250–259 (1999)Google Scholar
  4. 4.
    Bosman, S., Groenendaal, B., Findlater, J.-W., Visser, T., de Graaf, M., Markopoulos, P. : Gentleguide: an exploration of haptic output for indoors pedestrian guidance. In: International Conference on Mobile Human–Computer Interaction, pp. 358–362. Springer (2003)Google Scholar
  5. 5.
    Bradski, G., Kaehler, A.: Learning OpenCV: Computer Vision with the OpenCV Library. O’Reilly Media, Newton (2008)Google Scholar
  6. 6.
    Hough, P.V.C.: Method and means for recognizing complex patterns. US Patent 3,069,654, 18 Dec (1962)Google Scholar
  7. 7.
    Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. PAMI 8(6), 679–698 (1986)Google Scholar
  8. 8.
    Carmigniani, J., Furht, B.: Augmented reality: an overview. In: Handbook of Augmented Reality, pp. 3–46. Springer, New York (2011)Google Scholar
  9. 9.
    Caudell, T.P., Mizell, D.W.: Augmented reality: an application of heads-up display technology to manual manufacturing processes. In: Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences, 1992, vol. 2, pp. 659–669. IEEE (1992)Google Scholar
  10. 10.
    Douglas, D.H., Peucker, T.K.: Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Cartogr. Int. J. Geogr. Inf. Geovis. 10(2), 112–122 (1973)Google Scholar
  11. 11.
    Elvins, T.T.: Augmented reality: “the future’s so bright, i gotta wear (see-through) shades”. ACM SIGGRAPH Comput. Graph. 32(1), 11–13 (1998)Google Scholar
  12. 12.
    Fischer, X., Coutellier, D.: The interaction: a new way of designing. In: Research in Interactive Design, pp. 1–15. Springer, Paris (2006)Google Scholar
  13. 13.
    Fite-Georgel, P.: Is there a reality in industrial augmented reality? In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 201–210. IEEE (2011)Google Scholar
  14. 14.
    Flusser, J., Zitova, B., Suk, T.: Moments and Moment Invariants in Pattern Recognition. Wiley, Hoboken (2009)zbMATHGoogle Scholar
  15. 15.
    Gonzalez-Badillo, G., Medellin-Castillo, H.I., Lim, T.: Development of a haptic virtual reality system for assembly planning and evaluation. Procedia Technol. 7, 265–272 (2013)Google Scholar
  16. 16.
    Graham, R.L., Yao, F.F.: Finding the convex hull of a simple polygon. J. Algorithms 4(4), 324–331 (1983)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Guo, W., Ni, W., Chen, I.-M., Ding, Z.Q., Yeo, S.H.: Intuitive vibro-tactile feedback for human body movement guidance. In: 2009 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 135–140. IEEE (2009)Google Scholar
  18. 18.
    Hancock, P.A., Mercado, J.E., Merlo, J., Van Erp, J.B.F.: Improving target detection in visual search through the augmenting multi-sensory cues. Ergonomics 56(5), 729–738 (2013)Google Scholar
  19. 19.
    Henderson, S.J., Feiner, S.K: Augmented reality in the psychomotor phase of a procedural task. In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 191–200. IEEE (2011)Google Scholar
  20. 20.
    Hincapié, M., Caponio, A., Rios, H., Mendívil, E.G.: An introduction to augmented reality with applications in aeronautical maintenance. In: 2011 13th International Conference on Transparent Optical Networks (ICTON), pp. 1–4. IEEE (2011)Google Scholar
  21. 21.
    Ming-Kuei, H.: Visual pattern recognition by moment invariants. IRE Trans. Inf. Theory 8(2), 179–187 (1962)zbMATHGoogle Scholar
  22. 22.
    Huang, Z., Leng, J.: Analysis of Hu’s moment invariants on image scaling and rotation. In: 2010 2nd International Conference on Computer Engineering and Technology (ICCET), vol. 7, pp. V7–476. IEEE (2010)Google Scholar
  23. 23.
    Jain, R., Kasturi, R., Schunck, B.G.: Machine Vision. Computer Science Series. McGraw-Hill, New York (1995)Google Scholar
  24. 24.
    Jeon, S., Choi, S.: Haptic augmented reality: taxonomy and an example of stiffness modulation. Presence Teleoper. Virtual Environ. 18(5), 387–408 (2009)Google Scholar
  25. 25.
    Kitagawa, M., Yamamoto, T.: 3D puzzle guidance in augmented reality environment using a 3D desk surface projection. In: 2011 IEEE Symposium on 3D User Interfaces (3DUI), pp. 133–134. IEEE (2011)Google Scholar
  26. 26.
    Kovac, J., Peer, P., Solina, F.: Human skin color clustering for face detection, vol. 2. IEEE (2003)Google Scholar
  27. 27.
    Krig, S.: Computer Vision Metrics: Survey, Taxonomy, and Analysis. Springer, Berlin (2014)zbMATHGoogle Scholar
  28. 28.
    Lamberti, F., Manuri, F., Sanna, A., Paravati, G., Pezzolla, P., Montuschi, P.: Challenges, opportunities, and future trends of emerging techniques for augmented reality-based maintenance. IEEE Trans. Emerg. Top. Comput. 2(4), 411–421 (2014)Google Scholar
  29. 29.
    Lehtinen, V., Oulasvirta, A., Salovaara, A., Nurmi, P.: Dynamic tactile guidance for visual search tasks. In: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, pp. 445–452. ACM (2012)Google Scholar
  30. 30.
    Lepelley, M.C., Lejeune, L., Lestienne, F.: Study of tactile guidance of the hand in a pointing task by means of a vibrotactile device. Comput. Methods Biomech. Biomed. Eng. 12(S1), 169–170 (2009)Google Scholar
  31. 31.
    Luces, J.V.S., Okabe, K., Murao, Y., Hirata, Y.: A phantom-sensation based paradigm for continuous vibrotactile wrist guidance in two-dimensional space. IEEE Robot. Autom. Lett. 3(1), 163–170 (2018)Google Scholar
  32. 32.
    Mante, N., Weiland, J.D.: Visually impaired users can locate and grasp objects under the guidance of computer vision and non-visual feedback. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 1–4. IEEE (2018)Google Scholar
  33. 33.
    Nee, A.Y.C., Ong, S.K., Chryssolouris, G., Mourtzis, D.: Augmented reality applications in design and manufacturing. CIRP Ann. Manuf. Technol. 61(2), 657–679 (2012)Google Scholar
  34. 34.
    Nishihara, A., Okamoto Jr., J.: Object recognition in assembly assisted by augmented reality system. In: SAI Intelligent Systems Conference (IntelliSys), 2015, pp. 400–407. IEEE (2015)Google Scholar
  35. 35.
    Okamoto Jr., J., Nishihara, A.: Assembly Assisted by Augmented Reality (A3R), pp. 281–300. Springer, Berlin (2016)Google Scholar
  36. 36.
    Ong, S.K., Yuan, M.L., Nee, A.Y.C.: Augmented reality applications in manufacturing: a survey. Int. J. Prod. Res. 46(10), 2707–2742 (2008)zbMATHGoogle Scholar
  37. 37.
    Oron-Gilad, T., Downs, J.L., Gilson, R.D., Hancock, P.A.: Vibrotactile guidance cues for target acquisition. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 37(5), 993–1004 (2007)Google Scholar
  38. 38.
    Palmarini, R., Erkoyuncu, J.A., Roy, R., Torabmostaedi, H.: A systematic review of augmented reality applications in maintenance. Robot. Comput. Integr. Manuf. 49, 215–228 (2018)Google Scholar
  39. 39.
    Papagiannakis, G., Singh, G., Magnenat-Thalmann, N.: A survey of mobile and wireless technologies for augmented reality systems. Comput. Animat. Virtual Worlds 19(1), 3–22 (2008)Google Scholar
  40. 40.
    Pathomaree, N., Charoenseang, S.: Augmented reality for skill transfer in assembly task. In: IEEE International Workshop on Robot and Human Interactive Communication, 2005. ROMAN 2005, pp. 500–504. IEEE (2005)Google Scholar
  41. 41.
    Salonen, T., Sääski, J.: Dynamic and visual assembly instruction for configurable products using augmented reality techniques. In: Yan, X.T., Jiang, C., Eynard, B. (eds.) Advanced Design and Manufacture to Gain a Competitive Edge, pp. 23–32. Springer, London (2008)Google Scholar
  42. 42.
    Salonen, T., Sääski, J., Hakkarainen, M., Kannetis, T., Perakakis, M., Siltanen, S., Potamianos, A., Korkalo, O., Woodward, C.: Demonstration of assembly work using augmented reality. In: Proceedings of the 6th ACM International Conference on Image and Video Retrieval, pp. 120–123. ACM (2007)Google Scholar
  43. 43.
    Sergi, F., Accoto, D., Campolo, D., Guglielmelli, E.: Forearm orientation guidance with a vibrotactile feedback bracelet: on the directionality of tactile motor communication. In: 2008 2nd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics, pp. 433–438. IEEE (2008)Google Scholar
  44. 44.
    Siltanen, S., Hakkarainen, M., Korkalo, O., Salonen, T., Sääski, J., Woodward, C., Kannetis, T., Perakakis, M., Potamianos, A.: Multimodal user interface for augmented assembly. In: IEEE 9th Workshop on Multimedia Signal Processing, 2007. MMSP 2007, pp. 78–81. IEEE (2007)Google Scholar
  45. 45.
    Straub, M., Riener, A., Ferscha, A.: Route guidance with a vibro-tactile waist belt. In: 4th European Conference on Smart Sensing and Context (2009)Google Scholar
  46. 46.
    Syberfeldt, A., Danielsson, O., Holm, M., Wang, L.: Visual assembling guidance using augmented reality. Procedia Manuf. 1, 98–109 (2015)Google Scholar
  47. 47.
    Tan, H., Gray, R., Young, J.J., Taylor, R.: A haptic back display for attentional and directional cueing. J. Haptics Res. 3, 1–20 (2003)Google Scholar
  48. 48.
    Tang, A., Owen, C., Biocca, F., Mou, W.: Comparative effectiveness of augmented reality in object assembly. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 73–80. ACM (2003)Google Scholar
  49. 49.
    Teague, M.R.: Image analysis via the general theory of moments. JOSA 70(8), 920–930 (1980)MathSciNetGoogle Scholar
  50. 50.
    Treiber, M.A.: An Introduction to Object Recognition: Selected Algorithms for a Wide Variety of Applications. Advances in Computer Vision and Pattern Recognition. Springer, London (2010)zbMATHGoogle Scholar
  51. 51.
    Tumler, J., Doil, F., Mecke, R., Paul, G., Schenk, M., Pfister, E.A., Huckauf, A., Bockelmann, I., Roggentin, A.: Mobile augmented reality in industrial applications: approaches for solution of user-related issues. In: Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 87–90. IEEE Computer Society (2008)Google Scholar
  52. 52.
    Tzorakoleftherakis, E., Murphey, T.D., Scheidt, R.A.: Augmenting sensorimotor control using “goal-aware” vibrotactile stimulation during reaching and manipulation behaviors. Exp. Brain Res. 234, 1–12 (2016)Google Scholar
  53. 53.
    Valentini, P.P.: Natural interface for interactive virtual assembly in augmented reality using leap motion controller. Int. J. Interact. Des. Manuf. (IJIDeM) 12, 1–9 (2018)Google Scholar
  54. 54.
    Van Krevelen, D.W.F., Poelman, R.: A survey of augmented reality technologies, applications and limitations. Int. J. Virtual Real. 9(2), 1 (2010)Google Scholar
  55. 55.
    Wagemans, J., Elder, J.H., Kubovy, M., Palmer, S.E., Peterson, M.A., Singh, M., von der Heydt, R.: A century of gestalt psychology in visual perception: I. Perceptual grouping and figure–ground organization. Psychol. Bull. 138(6), 1172 (2012)Google Scholar
  56. 56.
    Wang, X., Ong, S.K., Nee, A.Y.C.: A comprehensive survey of augmented reality assembly research. Adv. Manuf. 4(1), 1–22 (2016)Google Scholar
  57. 57.
    Webel, S., Bockholt, U., Engelke, T., Gavish, N., Olbrich, M., Preusche, C.: An augmented reality training platform for assembly and maintenance skills. Robot. Auton. Syst. 61(4), 398–403 (2013)Google Scholar
  58. 58.
    Weber, B., Schätzle, S., Hulin, T., Preusche, C., Deml, B.: Evaluation of a vibrotactile feedback device for spatial guidance. In: 2011 IEEE on World Haptics Conference (WHC), pp. 349–354. IEEE (2011)Google Scholar
  59. 59.
    Wertheimer, M.: Laws of organization in perceptual forms. In: Ellis, W.D. (ed.) A source book of Gestalt Psychology. Harcourt Brace, New York (1923)Google Scholar
  60. 60.
    Wiedenmaier, S., Oehme, O., Schmidt, L., Luczak, H.: Augmented reality (AR) for assembly processes design and experimental evaluation. Int. J. Hum. Comput. Interact. 16(3), 497–514 (2003)Google Scholar
  61. 61.
    Yuan, M.L., Ong, S.K., Nee, A.Y.C.: Augmented reality for assembly guidance using a virtual interactive tool. Int. J. Prod. Res. 46(7), 1745–1767 (2008)zbMATHGoogle Scholar

Copyright information

© Springer-Verlag France SAS, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Mechanical EngineeringPolitecnico di MilanoMilanItaly
  2. 2.Design Engineering Research GroupUniversidad EAFITMedellínColombia

Personalised recommendations