Abstract
Aiming at the problems of single input mode and lack of naturalness in the assembly process of existing AR systems, a tracking registration method of mobile AR assembly system is proposed based on multi-quantity and multi-degree of freedom natural fingertip interaction. Firstly, the real-time and stable tracking of hand area in complex environment is realized based on the hand region tracking; secondly, the fingertip detection and recognition based on K-COS and parallel vector is used to improve the precision and stability of fingertip recognition; thirdly, the special movement track of fingertip is recognized based on improved DTW algorithm, which has strong compatibility and feature gradient transformation for complex fingertip trajectory recognition; finally, through the real-time transformation of projection relationship between fingertip and virtual object, the interaction between fingertip and virtual object is made more natural and realistic. The experimental results show that in the complex environment of background, illumination, scale and rotation, the precision of fingertip detection and recognition is about 93%, and the precision of fingertip motion template matching is about 91%. The translation error of the registration method based on visual feature recognition is reduced by about 100pix compared with fingertip tracking registration method, and the efficiency of mobile AR-guided assembly method is improved by about 24.77% compared with the traditional manual assisted assembly method. These data verifies the strong interaction and practicability of the fingertips based on the user's multi-quantity and multi-degree of freedom features in the process of mobile AR guided assembly.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akhtar N, Mian A (2018) Threat of adversarial attacks on deep learning in computer vision: a survey. J Latex Class Files 23(8):1–21. https://doi.org/10.1109/ACCESS.2018.2807385
Bailey T, Durrant-Whyte H (2006) Simultaneous localization and mapping (SLAM): part II. IEEE Robot Autom Mag 13(3):108–117. https://doi.org/10.1007/978-3-319-17885-1_101198
Bang H, Choi D, Wojdynski BW et al (2019) How the level of personalization affects the effectiveness of personalized ad messages: the moderating role of narcissism. Int J Advert 36(05):1–23. https://doi.org/10.1080/02650487.2019.1590069
Benbelkacem S, Belhocine M, Bellarbi A et al (2013) Augmented reality for photovoltaic pumping systems maintenance tasks. Renew Energy 55(4):428–437. https://doi.org/10.1016/j.renene.2012.12.043
Bolme DS, Beveridge JR, Draper BA et al (2010) Visual object tracking using adaptive correlation filters. In: IEEE conference on computer vision and pattern recognition, pp 2544–2550. https://doi.org/10.1109/CVPR.2010.5539960
Chen J, Li J-H, Yang S-H et al (2017) Weighted optimization-based distributed Kalman filter for nonlinear target tracking in collaborative sensor networks. IEEE Trans Cybern 47(11):3892–3905. https://doi.org/10.1109/TCYB.2016.2587723
Civera J, Davison AJ, Montiel JM (2008) Inverse depth parametrization for monocular SLAM. IEEE Trans Robot 24(5):932–945. https://doi.org/10.1109/TRO.2008.2003276
Dillow C (2019) BMW augmented reality glasses help average Joes make repairs. Pop Sci 3(09):258–264. https://doi.org/10.1007/s00464-021-08363-8
Durrant-Whyte H, Bailey T (2006) Simultaneous localization and mapping: part I. IEEE Robot Autom Mag 13(2):99–110. https://doi.org/10.1109/MRA.2006.1638022
Gao J-Y, Zhang T-Z, Yang X-S et al (2018a) P2T: part-to-target tracking via deep regression learning. IEEE Trans Image Process 27(6):3074–3086. https://doi.org/10.1109/TIP.2018.2813166
Gao X, An H, Chen W et al (2018b) A survey on mobile augmented reality visualization. J Comput-Aided Design Comput Graph 30(1):1–8
Garon M, Lalonde JF (2017) Deep 6-DOF tracking. IEEE Trans vis Comput Graph (S1941-0506) 23(11):2410–2418. https://doi.org/10.1109/TVCG.2017.2734599
Gray RM (2006) Toeplitz and circulant matrices: a review (foundations and trends(r) in communications and information theory). Now Publishers Inc. (ISBN: 1933019239)
Gui Z-W, Wang Y-T, Liu Y et al (2014) Study on the application of two-dimensional code in mobile augmented reality. J Comput-Aided Design Comput Graph 26(1):34–39. https://doi.org/10.3969/j.issn.1003-9775.2014.01.005
Guo K, Leng J-W (2019) A real-time fingertip detection algorithm combining curvature and circular features. China Encepap 7(29):235–255. https://doi.org/10.3969/j.issn.2095-2783.2019.03.014
Hang F-J, Dai G-Z, Peng X-L (2016) A survey on human-computer interaction in virtual reality. Scientia Sinica Informationis. 6(12):23–48
Hare S, Golodetz S, Saffari A et al (2015) Struck: structured output tracking with kernels. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2015.2509974
Henriques JF, Caseiro R, Martins P et al (2015) High-speed tracking with kernelized correlation filters. IEEE Trans Pattern Anal Mach Intell 37(3):583–596. https://doi.org/10.1109/TPAMI.2014.2345390
Huy DQ, Vietcheslav I, Lee GSG (2017) See-through and spatial augmented reality a novel framework for human-robot interaction. In: International conference on control automation and robotics (ICCAR), pp 719–726. https://doi.org/10.1109/ICCAR.2017.7942791
Ibañez R, Soria Á, Teyseyre A et al (2017) Approximate string matching: a lightweight approach to recognize gestures with Kinect. Pattern Recognit 62(14):73–86. https://doi.org/10.1016/j.patcog.2016.08.022
Jin H, Dong-qi H, Yi-neng C et al (2016) A survey on human–computer interaction in mixed reality. J Comput-Aided Design Comput Graph 11(6):869–880. https://doi.org/10.3969/j.issn.1003-9775.2016.06.002
Kalal Z, Mikolajczyk K, Matas J (2012) Tracking-learning-detection. IEEE Trans Pattern Anal Mach Intell 34(7):1409–1422. https://doi.org/10.1109/TPAMI.2011.239
Klein G, Murray D (2008) Parallel tracking and mapping for small AR workspaces. In: IEEE and ACM international symposium on mixed and augmented reality. ACM, pp 1–10. https://doi.org/10.1109/ISMAR.2007.4538852
Klopschitz M, Schall G, Schmalstieg D et al (2010) visual tracking for augmented reality. In: Indoor positioning and indoor navigation (IPIN), 2010 international conference on. IEEE, pp 1–4. https://doi.org/10.1109/IPIN.2010.5648274
Lan S (2018) Research on augmented reality aided technology for manual assembly process. Huazhong University of Science and Technology
Least-Squares R, Rifkin R, Yeo G et al (2003) Regularized least-squares classification. Acta Electron Sin 190(1):93–104. https://doi.org/10.1016/j.patcog.2008.07.010
Li Y, Yin D (2014) AR tracking and registration method based-on TLD algorithm. J Syst Simul 26(9):2062–2067
Li X, Cha Y-F, Zhang T-Z et al (2019) Survey of visual object tracking algorithms based on deep learning. J Image Graph 21(12):2057–2080
Liu P, Song Y (2020) Segmentation of sonar imagery using convolutional neural networks and Markov random field. Multidimens Syst Signal Process 31(1):1–27. https://doi.org/10.1007/s11045-019-00652-9
Ma L-L, Zhang J, Wang J-Z (2014) Modified CRF algorithm for dynamic hand gesture recognition. In: 2014 33rd Chinese control conference (CCC). IEEE, pp 4763–4767. https://doi.org/10.1109/ChiCC.2014.6895744
Mao Z, Yuan J-J, Wu Z-R et al (2014) Real-time compressive tracking based on online feature selection. Opt Precis Eng 22(3):730–736. https://doi.org/10.3788/OPE.20142203.0730
Masood S, Qureshi MP, Shah MB et al (2014) Dynamic time wrapping based gesture recognition. In: 2014 international conference on robotics and emerging allied technologies in engineering (ICREATE). IEEE, pp 205–210. https://doi.org/10.1109/iCREATE.2014.6828366
Meng Y, Ya-xuan HE, De-li Y et al (2018) AR code recognition algorithm based on arc correction filter. J Univ Sci Technol China 48(1):1–6
Mourtzis D, Zogopoulos M, Vlachou E (2017) Augmented reality application to support remote maintenance as a service in the robotics industry. Procedia CIRP 63:46–51. https://doi.org/10.1016/j.procir.2017.03.154
Naglot D, Kulkarni M (2016) Real-time sign language recognition using the leap motion controller. In: International conference on inventive computation technologies IEEE, pp 324–329. https://doi.org/10.1109/INVENTIVE.2016.7830097
Pang H-B, Li Z-B, Ding Y-D (2015) Dynamic gesture recognition based on gesture contour model of time series. Huanan Ligong Daxue Xuebao J South China Univ Technol (nat Sci) 43(1):140–146. https://doi.org/10.3969/j.issn.1000-565X.2015.01.022
Park KB, Kim M, Choi SH et al (2020) Deep learning-based smart task assistance in wearable augmented reality. Robot Comput-Integr Manuf 63:101887. https://doi.org/10.1016/j.rcim.2019.101887
Pei-Hsuan C, Po-Hsuan T, Kai-Ten F (2018) Interactive mobile augmented reality system for image and hand motion tracking. IEEE Trans Veh Technol 55(3):615–628. https://doi.org/10.1109/PIMRC.2010.5671921
Roy R, Sikdar D, Mahadevappa M et al (2018) A fingertip force prediction model for grasp patterns characterised from the chaotic behaviour of EEG. Med Biol Eng Comput 53(28):19–51. https://doi.org/10.1007/s11517-018-1833-0
Shi X-C, Ling H-B, Pang Y et al (2019) Rank-1 tensor approximation for high-order association in multi-target tracking. Int J Comput vis 127(8):1063–1083. https://doi.org/10.1007/s11263-018-01147-z
Shim J, Yang Y, Kang N et al (2016) Gesture-based interactive augmented reality content authoring system using HMD. Virtual Real 20(1):57–69. https://doi.org/10.29121/ijoest.v5.i2.2021.162
Singh M, Chawla M, Singh R et al (2020) Disguised faces in the wild 2019. In: 2019 IEEE/CVF international conference on computer vision workshop (ICCVW). IEEE. https://doi.org/10.1109/ICCVW.2019.00067
Tang J-R, Cheng H, Zhao Y et al (2018) Structured dynamic time warping for continuous hand trajectory gesture recognition. Pattern Recognit 80:21–31. https://doi.org/10.1016/j.patcog.2018.02.011
Tran D, Bourdev L, Fergus R et al (2015) Learning spatiotemporal features with 3D convolutional networks. In: IEEE international conference on computer vision. IEEE, pp 4489–4497. https://doi.org/10.1109/ICCV.2015.510
Wadosky KM, Shourideh M, Goodrich DW et al (2019) Riluzole induces AR degradation via endoplasmic reticulum stress pathway in androgen dependent and castration-resistant prostate cancer cells. Prostate 79(2):140–146. https://doi.org/10.1002/pros.23719
Wang G-L, Tang Z (2017) Research on teaching methods based on mobile new augmented reality technology. Chin J Pedag S1:125–127
Wei SE, Ramakrishna V, Kanade T et al (2016) Convolutional pose machines. In: IEEE conference on computer vision and pattern recognition. IEEE, pp 4724–4732. https://doi.org/10.1109/CVPR.2016.511
Wu Y, Lim J, Yang MH (2013) Online object tracking: a benchmark. In: IEEE conference on computer vision and pattern recognition, pp 2411–2418. https://doi.org/10.1109/CVPR.2013.312
Xie Y-F, Huang Y, Song TL (2017) Iterative joint integrated probabilistic data association filter for multiple-detection multiple-target tracking. Digit Signal Process 72(3):232–243. https://doi.org/10.1016/j.dsp.2017.10.020
Xing-jian S, Chen Z-R, Wang H et al (2015) Convolutional LSTM network: a machine learning approach for precipitation nowcasting. Neural Inf Process Syst 1:802–810. https://doi.org/10.1007/978-3-319-21233-3_6
Yang S, Carlson JR, Chen S (2020) How augmented reality affects advertising effectiveness: the mediating effects of curiosity and attention toward the Ad. J Retail Consum Serv 54(3):112–119. https://doi.org/10.1016/j.jretconser.2019.102020
Yi P, Liu Y-C, Shi Y-J et al (2018) Design and development of engineering graphics teaching system on mobile devices based on the augmented reality technology. J Graph 39(6):1207–1213
Yin H-P, Chen B, Chai Y et al (2016) Vision-based object detection and tracking: a review. Acta Automatica Sinica 42(10):1466–1489. https://doi.org/10.16383/j.aas.2016.c150823
Zhang Z (2012) Microsoft Kinect sensor and its effect. IEEE Multimed 19(2):4–10. https://doi.org/10.1109/MMUL.2012.24
Zhang XX (2018) Application of augmented reality technology based on edge tracking in mechanical maintenance. Beijing University of Posts and Telecommunications
Zhang H-Y, Li C-F (2017) Compressive tracking algorithm combining online feature selection with covariance matrix. Opt Precis Eng 25(4):1501–1509. https://doi.org/10.3788/OPE.20172504.1051
Zhang G-L, Wu Y-X, Wang Z-N et al (2015) Research on augmented reality system modeling and registration error based on simple visual marker. Comput Sci 42(6):299–302. https://doi.org/10.11896/j.issn.1002-137X.2015.6.063
Zhang J, Henein M, Mahony R et al (2020) Robust ego and object 6-DoF motion estimation and tracking. arXiv 07:106–113. https://doi.org/10.1109/IROS45743.2020.9341552
Acknowledgements
This work was financially supported by the National Natural Science Foundation of China (62067006), The Ministry of education of Humanities and Social Science project(21YJC880085), Gansu Province Science and technology plan project (20JR10RA250,21JR7RA713,21YF5FA009), Gansu University Innovation Fund Project (2021B-092), Project supported by the Young Scholars Science Foundation of Lanzhou Jiaotong University (2021006), Open fund project of Key Laboratory of four power BIM engineering and intelligent application railway industry(BIMKF-2021-05), and China University industry university research innovation fund.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Jiu, Y., Jianguo, W., Yangping, W. et al. Fingertip interactive tracking registration method for AR assembly system. Adv. in Comp. Int. 2, 19 (2022). https://doi.org/10.1007/s43674-021-00025-5
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s43674-021-00025-5