Skip to main content

Advertisement

SpringerLink
Log in

Machining process-oriented monitoring method based on digital twin via augmented reality

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

The change of size, surface roughness, residual stress, and so on profoundly influence the final machining quality of complex mechanical products. Digital twin machining technology can ensure machining quality by observing the machining process in real time. However, the current digital twin systems mainly adopt the display method of virtual-real separation. It leads to transmitting the useful processing information to the on-site technicians ineffectively, limiting the digital twin system to help field processing. The monitoring technology on the machining process by augmented reality based on the digital twin machining system is proposed to deal with this problem. Firstly, the augmented reality dynamic multi-view is constructed based on multi-source heterogeneous data. Secondly, the augmented reality is integrated into the real-time monitoring of the intermediate process of complex products to promote cooperation among the operators and the digital twin machining system. It can avoid irreparable errors when the finished product is nearly completed. Finally, the effectiveness and feasibility of the proposed method will be verified by a monitoring application case.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5.
Fig. 6
Fig. 7.
Fig. 8.
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Abbreviations

P :

The processing procedure

p i :

One process of P

T i :

The time required to execute a process

k :

A segment of the process pi

t :

The time segment in process

P − St i():

The real state of the product in the physical world

DTP − St i(t, k):

The virtual state of the product in twin world

M − St i(t, k):

The state of machine tool in the physical world

T − St i(t, k):

The state of the tool in the physical world

DTMM :

Digital twin mimic model

GeoM :

The model contains the geometric information of the product

PhyM :

The model contains the physical information of the product

ConM :

The model contains the contextual information of the product

ARDMV :

The augmented reality dynamic multi-view

ARVO :

The augmented reality view object

VO :

View object

VO f :

The fixed view of ARDMV

VO s :

The selected view of ARDMV

VO r :

The recommended view of ARDMV

r p,v :

The rating of the view v in process p

P v :

The view

\( \overline{{\boldsymbol{r}}_{\boldsymbol{v}}} \) :

The average score of the view

w(p, p′):

The similarity of the process between p and p′

V p :

The view set of the process p

\( {\boldsymbol{p}}_{\boldsymbol{p},\boldsymbol{v}}^{\boldsymbol{c}} \) :

The recommended score of view v in a process p

C i :

The context information of a dimension

Pos o :

The initial posture

Pos i :

The current attitude estimation

Pos f :

The predicted posture

References

  1. Tao F, Qi QL (2019) Make more digital twins. Nature 573:490–491. https://doi.org/10.1038/d41586-019-02849-1

    Article  Google Scholar 

  2. Wang J, Xu C, Zhang J, Bao J, Zhong R (2019) A collaborative architecture of the industrial internet platform for manufacturing systems. Robot Comput Integr Manuf 61:1018–1054. https://doi.org/10.1016/j.rcim.2019.101854

    Article  Google Scholar 

  3. Tao J, Qin C, Xiao D, Shi H, Ling X, Li B, Liu C (2020) Timely chatter identification for robotic drilling using a local maximum synchrosqueezing-based method. J Intell Manuf 31(5):1243–1255. https://doi.org/10.1007/s10845-019-01509-5

    Article  Google Scholar 

  4. Qin C, Tao J, Shi H, Xiao D, Li B, Liu C (2020) A novel Chebyshev-wavelet-based approach for accurate and fast prediction of milling stability. Precis Eng 62(1):244–255. https://doi.org/10.1016/j.precisioneng.2019.11.016

    Article  Google Scholar 

  5. Jin Y, Qin C, Huang Y, Liu C (2021) Actual bearing compound fault diagnosis based on active learning and decoupling attentional residual network. Measurement 173:108500. https://doi.org/10.1016/j.measurement.2020.108500 

  6. Qin C, Shi G, Tao J, Yu H, Jin Y, Lei J, Liu C (2021) Precise cutterhead torque prediction for shield tunneling machines using a novel hybrid deep neural network. Mechanical Systems and Signal Processing 151:107386. https://doi.org/10.1016/j.ymssp.2020.107386 

  7. Grieves M (2014) Digital twin: manufacturing excellence through virtual factory replication. Whitepaper. https://zenodo.org/record/1493930. Accessed 29 October 2020

  8. Lu Y, Liu C, Wang K, Huang H, Xu X (2020) Digital twin-driven smart manufacturing: connotation, reference model, applications and research issues. Robot Comput Integr Manuf 61:101837. https://doi.org/10.1016/j.rcim.2019.101837

    Article  Google Scholar 

  9. Brenner B, Hummel V (2017) Digital Twin as Enabler for an Innovative Digital Shopfloor Management System in the ESB Logistics Learning Factory at Reutlingen-University. Procedia Manuf 9:198–205. https://doi.org/10.1016/j.promfg.2017.04.039

    Article  Google Scholar 

  10. Tao F, Zhang M (2017) Digital twin shop-floor: a new shop- floor paradigm towards smart manufacturing. IEEE Access 5:20418–20427. https://doi.org/10.1109/ACCESS.2017.2756069

    Article  Google Scholar 

  11. Zhuang CB, Liu JH, Xiong H (2018) Digital twin-based smart production management and control framework for the complex product assembly shop-floor. Int J Adv Manuf Technol 96(1-4):1149–1163. https://doi.org/10.1007/s00170-018-1617-6

    Article  Google Scholar 

  12. Zhang H, Liu Q, Chen X, Zhang D, Leng JW (2017) A digital twin-based approach for designing and decoupling of hollow glass production line. IEEE Access 5:26901–26911. https://doi.org/10.1109/ACCESS.2017.2766453

    Article  Google Scholar 

  13. Liu Q, Zhang H, Leng JW (2019) Digital twin-driven rapid individualised designing of automated flow-shop manufacturing system. Int J Prod Res, 3309-3319. doi:https://doi.org/10.1080/00207543.2018.1471243.

  14. Grieves M, Vickers J (2017) Digital twin: mitigating unpredictable, undesirable emergent behavior in complex systems. Transdiscip Perspect Complex Syst Berlin, Germany: Springer-Verlag. 85-113 https://doi.org/10.1007/978-3-319-38756-7_4

  15. Luo WC, Hu TL, Zhu WD (2018) Digital twin modeling method for CNC machine tool. IEEE 15th International Conference on Networking, Sensing and Control (ICNSC). https://doi.org/10.1109/ICNSC.2018.8361285

  16. Rosen R, Wichert GV, Lo G (2015) About the importance of autonomy and digital twins for the future of manufacturing. Ifac Papersonline 48(3):567–572. https://doi.org/10.1016/j.ifacol.2015.06.141

    Article  Google Scholar 

  17. Qi Q, Tao F, Zuo Y, Zhao D (2018) Digital twin service towards smart manufacturing. Procedia CIRP 72:237–242. https://doi.org/10.1016/j.procir.2018.03.103

    Article  Google Scholar 

  18. Rok V, John AE, Peter B, Rajkumar R (2018) Digital twins: understanding the added value of integrated models for through-life engineering services. Procedia Manuf 16:139–146. https://doi.org/10.1016/j.promfg.2018.10.167

    Article  Google Scholar 

  19. Tao F, Liu WR, Zhang M (2019) Five-dimension digital twin model and its ten applications. Comput Integr Manuf Syst 25(1):1–18. https://doi.org/10.13196/j.cims.2019.01.001

    Article  Google Scholar 

  20. Liu SM, Bao JS, Lu YQ, Li J, Lu SY, Sun XM (2020) Digital twin modeling method based on biomimicry for machining aerospace components. J Manuf Syst. https://doi.org/10.1016/j.jmsy.2020.04.014

  21. Hao X, Li Y, Cheng Y, Liu C, Xu K, Tang K (2020) A time-varying geometry modeling method for parts with deformation during machining process. J Manuf Syst 55:15–29. https://doi.org/10.1016/j.jmsy.2020.02.002

    Article  Google Scholar 

  22. Luo W, Hu T, Zhang C, Wei Y (2019) Digital twin for CNC machine tool: modeling and using strategy. J Ambient Intell Humaniz Comput 10(3):1129–1140. https://doi.org/10.1007/s12652-018-0946-5

    Article  Google Scholar 

  23. Tong X, Liu Q, Pi S, Xiao Y (2020) Real-time machining data application and service based on IMT digital twin. J Intell Manuf 31(5):1113–1132. https://doi.org/10.1007/s10845-019-01500-0

    Article  Google Scholar 

  24. Urbina Coronado PD, Lynn R, Louhichi W, Parto M, Wescoat E, Kurfess T (2018) Part data integration in the Shop Floor Digital Twin: mobile and cloud technologies to enable a manufacturing execution system. J Manuf Syst 48:25–33. https://doi.org/10.1016/j.jmsy.2018.02.002

    Article  Google Scholar 

  25. Zhao P, Liu J, Jing X, Tang M, Sheng S, Zhou H, Liu X (2020) The modeling and using strategy for the digital twin in process planning. IEEE Access 8:41229–41245. https://doi.org/10.1109/ACCESS.2020.2974241

    Article  Google Scholar 

  26. Kong T, Hu T, Zhou T, Ye Y (2020) Data construction method for the applications of workshop digital twin system. J Manuf Syst. https://doi.org/10.1016/j.jmsy.2020.02.003

  27. Liu J, Zhou H, Liu X, Tian G, Wu M, Cao L, Wang W (2019) Dynamic evaluation method of machining process planning based on digital twin. IEEE Access 7:19312–19323. https://doi.org/10.1109/ACCESS.2019.2893309

    Article  Google Scholar 

  28. Liu J, Zhou H, Tian G, Liu X, Jing X (2019) Digital twin-based process reuse and evaluation approach for smart process planning. Int J Adv Manuf Technol 100(5-8):1619–1634. https://doi.org/10.1007/s00170-018-2748-5

    Article  Google Scholar 

  29. Zhou G, Zhang C, Li Z, Ding K, Wang C (2020) Knowledge-driven digital twin manufacturing cell towards intelligent manufacturing. Int J Prod Res 58(4):1034–1051. https://doi.org/10.1080/00207543.2019.1607978

    Article  Google Scholar 

  30. Li X (2020) Research and implementation of virtual monitoring system for machine tools process based on digital twin. Dissertation, University of Electronic Science and technology

  31. Cao X, Zhao G, Xiao W (2020) Digital twin–oriented real-time cutting simulation for intelligent computer numerical control machining. P I Mech Eng B-J Eng. https://doi.org/10.1177/0954405420937869

  32. Zhu L, Li H, Liang W, Wang W (2015) A web-based virtual CNC turn-milling system. Int J Adv Manuf Technol 78(1-4):99–113. https://doi.org/10.1007/s00170-014-6649-y

    Article  Google Scholar 

  33. Liu C, Vengayil H, Zhong RY, Xu X (2018) A systematic development method for cyber-physical machine tools. J Manuf Syst 48:13–24. https://doi.org/10.1016/j.jmsy.2018.02.001

    Article  Google Scholar 

  34. Ma X, Tao F, Zhang M, Wang T, Zuo Y (2019) Digital twin enhanced human-machine interaction in product lifecycle. Procedia CIRP 83:789–793. https://doi.org/10.1016/j.procir.2019.04.330

    Article  Google Scholar 

  35. Li W, Wang JF, Lan S, Li SQ, Jiao SC, W M (2019) Content authoring of augmented reality assembly process. Comput Integr Manuf Syst 25(07):1676–1684. https://doi.org/10.13196/j.cims.2019.07.008

    Article  Google Scholar 

  36. Cheng Y, Huang R, Jiang JF, Chen ZM, Xu T (2019) Product manufacturing information transmission method based on augmented reality. J Comput Aid Design Comput Graph 31(5):859–868. https://doi.org/10.3724/SP.J.1089.2019.17370

    Article  Google Scholar 

  37. Fang W, An ZW (2020) Research on intelligent order picking method based on wearable augmented reality. Comput. Integr Manuf Syst :1-14. http://kns.cnki.net/kcms/detail/11.5946.TP.20200520.1729.018.html

  38. Kokkas A, Vosniakos G (2019) An augmented reality approach to factory layout design embedding operation simulation. Int J Interact Des Manuf 13(3):1061–1071. https://doi.org/10.1007/s12008-019-00567-6

    Article  Google Scholar 

  39. Ceruti A, Liverani A, Bombardi T (2017) Augmented vision and interactive monitoring in 3D printing process. Int J Interact Des Manuf 11(2):385–395. https://doi.org/10.1007/s12008-016-0347-y

    Article  Google Scholar 

  40. Liu C, Cao S, Tse W, Xu X (2017) Augmented reality-assisted intelligent window for cyber-physical machine tools. J Manuf Syst 44:280–286. https://doi.org/10.1016/j.jmsy.2017.04.008

    Article  Google Scholar 

  41. Zhou J, Zhou Y, Wang B, Zang J (2019) Human–cyber–physical systems (HCPSs) in the context of new-generation intelligent manufacturing. Engineering-PRC 5(4):624–636. https://doi.org/10.1016/j.eng.2019.07.015

    Article  Google Scholar 

  42. Wang Y, Zhang S, Yang S, He W, Bai X, Zeng Y (2017) A LINE-MOD-based marker less tracking approach for AR applications. Int J Adv Manuf Technol 89(5-8):1699–1707. https://doi.org/10.1007/s00170-016-9180-5

    Article  Google Scholar 

Download references

Availability of data and materials

The datasets used or analyzed during the current study are available from the corresponding author on reasonable request.

Funding

This work is financially supported by the National Key Research and Development Plan of China (Grant 2019YFB1706300), in part by the Fundamental Research Funds for the Central Universities and Graduate Student Innovation Fund of Donghua University (Grant No. CUSF-DH-D-2020056), in part by the Fundamental Research Funds for the Central Universities (No. 2232019D3-32), and in part by Shanghai Sailing Program (19YF1401600).

Author information

Authors and Affiliations

  1. College of Mechanical Engineering, Donghua University, Shanghai, 201620, China

    Shimin Liu, Shanyu Lu, Jie Li, Xuemin Sun & Jinsong Bao

  2. Department of Mechanical Engineering, The University of Auckland, Auckland, 1010, New Zealand

    Yuqian Lu

Authors
  1. Shimin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shanyu Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jie Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xuemin Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuqian Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jinsong Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Shimin Liu: Conceptualization, methodology, software, writing—original draft, writing—review and editing, resources, data curation, and visualization

Shanyu Lu: Conceptualization, methodology, writing—review and editing, and supervision

Jie Li: Conceptualization, methodology, and supervision

Yuqian Lu: Conceptualization, methodology, and supervision

Xuemin Sun: Software, visualization, and validation

Jinsong Bao: Supervision, project administration, and funding acquisition

Corresponding author

Correspondence to Jinsong Bao.

Ethics declarations

Ethical approval

The author confirms that the manuscript is not submitted to more than one journal for simultaneous consideration, that the submitted work is original and is not published elsewhere in any form or language, that the study is not split up into several parts to increase the quantity of submissions and submitted to various journals or to one journal over time, and that results are presented clearly, honestly, and without fabrication, falsification, or inappropriate data manipulation. Authors have adhered to discipline-specific rules for acquiring, selecting, and processing data.

Consent to participate

Participants in the test clearly understand the risk and signed the relevant informed consent to participate in the experiment.

Consent to publish

The author confirms that the work described has not been published before (except in the form of an abstract or as part of a published lecture, review, or thesis); that it is not under consideration for publication elsewhere; that its publication has been approved by all co-authors, if any; and that its publication has been approved (tacitly or explicitly) by the responsible authorities at the institution where the work is carried out.

The author agrees to the publication in the Journal indicated below and also to the publication of the article in English by Springer in Springer’s corresponding English-language journal.

The copyright to the English-language article is transferred to Springer effective if and when the article is accepted for publication. The author warrants that his/her contribution is original and that he/she has full power to make this grant. The author signs for and accepts responsibility for releasing this material on behalf of any and all co-authors. The copyright transfer covers the exclusive right to reproduce and distribute the article, including reprints, translations, photographic reproductions, microform, electronic form (offline, online), or any other reproductions of similar nature.

Competing interests

The authors of this paper have no affiliations with or involvement in any organization or entity with any financial interest (such as honoraria; educational grants; participation in speakers’ bureaus; membership, employment, consultancies, stock ownership, or other equity interest; and expert testimony or patent-licensing arrangements), or non-financial interest (such as personal or professional relationships, affiliations, knowledge, or beliefs) in the subject matter or materials discussed in this manuscript.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, S., Lu, S., Li, J. et al. Machining process-oriented monitoring method based on digital twin via augmented reality. Int J Adv Manuf Technol 113, 3491–3508 (2021). https://doi.org/10.1007/s00170-021-06838-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-021-06838-5

Keywords

Search

Navigation