Skip to main content
SpringerLink
Log in

A rapid generation method of models in machining processes for real-time human–machine interaction with virtual-real fusion

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

The intelligent service of the digital twin machine tool provides convenience for the human operation interaction in the machine tool, and the real-time operation interaction between the human and the machine tool has high requirements for the real-time machining model simulation algorithm. Firstly, this paper proposes a simulation method based on the combination of augmented reality (AR) and digital twin of machine tool machining virtual-real fusion. Secondly, to improve the real-time interoperability between human and machine tools in the AR virtual-real fusion machining process, this paper proposes a fast Dexel model generation method based on binary tree space segmentation. The method is based on the 3D model of the workpiece preprocessing to generate a one-way Dexel model of the binary tree storage structure, using the tool and the workpiece overlap envelope in the binary tree structure to determine the interference region, and ultimately calculating the intersection line between the one-way generation line and the tool geometry to get the model of the workpiece after cutting. By analyzing the real-time performance of the algorithm, the algorithm satisfies the simulation calculation of Dexel models of different scales. Finally, through the example of real-time interactive operation between human and machine tool AR, the results show that the average display frame rate of this algorithm in the machining process reaches 55–60 frames, and the parameter error between the model after virtual machining and the actual machining model is within 1.3%. At the same time, 100 people were randomly selected to carry out AR interaction training, and the real-time performance experience of AR virtual-real fusion machining was comprehensively evaluated, and the results showed that the system can meet the real-time demand of interaction operation of most participants.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability

The authors confirm that the data supporting the findings of this study are available within the article.

Code availability

This paper (A rapid generation method of models in machining processes for real-time human–machine interaction with virtual-real fusion) is not applicable.

References

  1. Tong X, Liu Q, Pi S, Xiao Y (2020) Real-time machining data application and service based on IMT digital twin. J Intell Manuf 31:1113–1132. https://doi.org/10.1007/s10845-019-01500-0

    Article  Google Scholar 

  2. Zhou Q, Yan P, Xin Y (2017) Research on a knowledge modelling methodology for fault diagnosis of machine tools based on formal semantics[J]. Adv Eng Inform 32:92–112. https://doi.org/10.1016/j.aei.2017.01.002

    Article  Google Scholar 

  3. Gao Z, Zhang D (2015) Performance analysis, mapping, and multiobjective optimization of a hybrid robotic machine tool. IEEE Trans Ind Electron 62:423–433. https://doi.org/10.1109/TIE.2014.2327008

    Article  Google Scholar 

  4. Shahriar MR, Al Sunny SN, Liu X, Leu MC, Hu L, & Nguyen NT (2018). MTComm based virtualization and integration of physical machine operations with digital-twins in cyber-physical manufacturing cloud. In 2018 5th IEEE International Conference on Cyber Security and Cloud Computing (CSCloud)/2018 4th IEEE International Conference on Edge Computing and Scalable Cloud (EdgeCom) (pp. 46–51). IEEE. https://doi.org/10.1109/CSCloud/EdgeCom.2018.00018.

  5. Li J, Yao Y, Wu J (2011) CNC Partner: a novel training system for NC machining. Comp Applic In Engineering 19:466–474. https://doi.org/10.1002/cae.20326

    Article  Google Scholar 

  6. Baicun WANG, Yuan XUE, Jianlin YAN, Xiaoying Y, Yuan Z (2020) Human-centered intelligent manufacturing: overview and perspectives. Strategic Study of CAE 22(4):139–146. https://doi.org/10.15302/J-SSCAE-2020.04.020

    Article  Google Scholar 

  7. Gorecky D, Schmitt M, Loskyll M, Zühlke D (2014) Human-machine-interaction in the industry 4.0 era. In: 2014 12th IEEE International Conference on Industrial Informatics (INDIN). pp 289–294. https://doi.org/10.1109/INDIN.2014.6945523.

  8. Yang X, Ran Y, Zhang G, Wang H, Mu Z, Zhi S (2022) A digital twin-driven hybrid approach for the prediction of performance degradation in transmission unit of CNC machine tool. Robot Comput Integr Manuf 73:102230. https://doi.org/10.1016/j.rcim.2021.102230

    Article  Google Scholar 

  9. Tao F, Zhang M, Cheng J, Qi Q (2017) Digital twin workshop: a new paradigm for future workshop. Comput Integr Manuf Syst 23:1–9. https://doi.org/10.13196/j.cims.2017.01.001

    Article  Google Scholar 

  10. Liu C, Hong X, Zhu Z, Xu X (2018) Machine tool digital twin: modelling methodology and applications. https://www.researchgate.net/publication/332606463

  11. Luo W, Hu T, Ye Y, Zhang C, Wei Y (2020) A hybrid predictive maintenance approach for CNC machine tool driven by digital twin. Robot Comput Integr Manuf 65:101974. https://doi.org/10.1016/j.rcim.2020.101974

    Article  Google Scholar 

  12. Wei Y, Hu T, Zhou T, Ye Y, Luo W (2021) Consistency retention method for CNC machine tool digital twin model. J Manuf Syst 58:313–322. https://doi.org/10.1016/j.jmsy.2020.06.002

    Article  Google Scholar 

  13. Liu C, Xu X, Peng Q, Zhou Z (2018) MTConnect-based cyber-physical machine tool: a case study. Procedia CIRP 72:492–497. https://doi.org/10.1016/j.procir.2018.03.059

    Article  Google Scholar 

  14. Liu C, Zhu H, Tang D, Nie Q, Zhou T, Wang L, Song Y (2022) Probing an intelligent predictive maintenance approach with deep learning and augmented reality for machine tools in IoT-enabled manufacturing. Robot Comput Integr Manuf 77:102357. https://doi.org/10.1016/j.rcim.2022.102357

    Article  Google Scholar 

  15. Luo W, Hu T, Zhang C, Wei Y (2019) Digital twin for CNC machine tool: modeling and using strategy. J Ambient Intell Human Comput 10:1129–1140. https://doi.org/10.1007/s12652-018-0946-5

    Article  Google Scholar 

  16. Jiang H, Qin S, Fu J, Zhang J, Ding G (2021) How to model and implement connections between physical and virtual models for digital twin application. J Manuf Syst 58:36–51. https://doi.org/10.1016/j.jmsy.2020.05.012

    Article  Google Scholar 

  17. P Janda 2018 Mechatronic concept of heavy machine tools.https://doi.org/10.2507/29th.daaam.proceedings.093

  18. Uhlemann TH-J, Lehmann C, Steinhilper R (2017) The digital twin: realizing the cyber-physical production system for industry 4.0. Procedia Cirp 61:335–340. https://doi.org/10.1016/j.procir.2016.11.152

    Article  Google Scholar 

  19. Cichon T, Roßmann J (2018) Digital twins: assisting and supporting cooperation in human-robot teams. In: 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV). IEEE, pp 486–491. https://doi.org/10.1109/ICARCV.2018.8580634.

  20. Park K-B, Kim M, Choi SH, Lee JY (2020) Deep learning-based smart task assistance in wearable augmented reality. Robot Comput Integr Manuf 63:101887. https://doi.org/10.1016/j.rcim.2019.101887

    Article  Google Scholar 

  21. Choi SH, Park KB, Roh DH, Lee JY, Mohammed M, Ghasemi Y, Jeong H (2022) An integrated mixed reality system for safety-aware human-robot collaboration using deep learning and digital twin generation. Robot Comput Integr Manuf 73:102258. https://doi.org/10.1016/j.rcim.2021.102258

    Article  Google Scholar 

  22. Eswaran M, Bahubalendruni MR (2022) Challenges and opportunities on AR/VR technologies for manufacturing systems in the context of industry 4.0: a state of the art review. J Manuf Syst 65:260–278. https://doi.org/10.1016/j.jmsy.2022.09.016

    Article  Google Scholar 

  23. Geng R, Li M, Hu Z, Han Z, Zheng R (2022) Digital Twin in smart manufacturing: remote control and virtual machining using VR and AR technologies. Struct Multidisc Optim 65(11):321. https://doi.org/10.1007/s00158-022-03426-3

    Article  Google Scholar 

  24. Zhang J, Ong SK, Nee AYC (2012) Design and development of an in situ machining simulation system using augmented reality technology. Proc CIRP 3:185–190. https://doi.org/10.1016/j.procir.2012.07.033

    Article  Google Scholar 

  25. Liu S, Lu S, Li J, Sun X, Lu Y, Bao J (2021) Machining process-oriented monitoring method based on digital twin via augmented reality. Int J Adv Manuf Technol 113:3491–3508. https://doi.org/10.1007/s00170-021-06838-5

    Article  Google Scholar 

  26. Zhu Z, Liu C, Xu X (2019) Visualisation of the digital twin data in manufacturing by using augmented reality. Procedia Cirp 81:898–903. https://doi.org/10.1016/j.procir.2019.03.223

    Article  Google Scholar 

  27. Syberfeldt A, Danielsson O, Holm M, Wang L (2015) Visual assembling guidance using augmented reality. Proc Manuf 1:98–109. https://doi.org/10.1016/j.promfg.2015.09.068

    Article  Google Scholar 

  28. Liu C, Cao S, Tse W, Xu X (2017) Augmented reality-assisted intelligent window for cyber-physical machine tools. J Manuf Syst 44:280–286. https://doi.org/10.1016/j.jmsy.2017.04.008

    Article  Google Scholar 

  29. Kadir AA, Xu X (2011) Towards high-fidelity machining simulation. J Manuf Syst 30:175–186. https://doi.org/10.1016/j.jmsy.2011.04.004

    Article  Google Scholar 

  30. Sun YJ, Yan C, Wu SW, Gong H, Lee CH (2018) Geometric simulation of 5-axis hybrid additive-subtractive manufacturing based on tri-Dexel model. Int J Adv Manuf Technol 99:2597–2610. https://doi.org/10.1007/s00170-018-2577-6

    Article  Google Scholar 

  31. Inui M, Umezu N (2010) Implementation of a 5-axis milling simulation system using triple Dexel models. J Jpn Soc Precis Eng 76:361–366. https://doi.org/10.2493/jjspe.76.361

    Article  Google Scholar 

  32. Cao X, Zhao G, Xiao W (2022) Digital twin-oriented real-time cutting simulation for intelligent computer numerical control machining. Proc Inst Mech Eng Part B-J Eng Manuf 236:5–15. https://doi.org/10.1177/0954405420937869

    Article  Google Scholar 

  33. Schönemann L, Riemer O, Karpuschewski B, Schreiber P, Klemme H, Denkena B (2022) Digital surface twin for ultra-precision high performance cutting. Precis Eng 77:349–359. https://doi.org/10.1016/j.precisioneng.2022.06.010

    Article  Google Scholar 

  34. Yu H, Wu D, Xu H (2024) Reconstruction algorithm for complex Dexel models based on composite block partition. J Comput Inf Sci Eng 24:041004. https://doi.org/10.1115/1.4063955

    Article  Google Scholar 

  35. Altintas Y, Kersting P, Biermann D, Budak E, Denkena B, Lazoglu I (2014) Virtual process systems for part machining operations. CIRP Ann 63(2):585–605. https://doi.org/10.1016/j.cirp.2014.05.007

    Article  Google Scholar 

  36. Radkowski R (2016) Object tracking with a range camera for augmented reality assembly assistance. J Comput Inf Sci Eng 16:011004. https://doi.org/10.1115/1.4031981

    Article  Google Scholar 

  37. Macedo MC, Apolinario AL (2021) Occlusion handling in augmented reality: past, present and future. IEEE Trans Visual Comput Graphics 29:1590–1609. https://doi.org/10.1109/TVCG.2021.3117866

    Article  Google Scholar 

  38. Dambly V, Rivière-Lorphèvre É, Ducobu F, Verlinden O (2024) Tri-Dexel-based cutter-workpiece engagement: computation and validation for virtual machining operations. Int J Adv Manuf Technol 131:623–635. https://doi.org/10.1007/s00170-023-10950-z

    Article  Google Scholar 

  39. Kaufeld M, Mundt M, Forst S, Hecht H (2022) Optical see-through augmented reality can induce severe motion sickness. Displays 74:102283. https://doi.org/10.1016/j.displa.2022.102283

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Key R&D Program of China (2018YFB1701303). Author Hanzhong Xu has received research support from the Shanghai Electrical Apparatus Research Institute.

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, Shanghai Jiaotong University, Shanghai, 200240, China

    Hanzhong Xu, Dianliang Wu, Yu Zheng, Haiwen Yu, Qihang Yu & Kai Zou

Authors
  1. Hanzhong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dianliang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haiwen Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qihang Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kai Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Hanzhong Xu: conceptualization, experimental, writing—original draft; Dianliang Wu: supervision, financial support; Yu Zheng: review and editing; Haiwen Yu: experimental; Qihang Yu: review and editing; Kai Zou: conceptualization.

Corresponding author

Correspondence to Hanzhong Xu.

Ethics declarations

Ethics approval

The authors have adhered to discipline-specific rules for acquiring, selecting, and processing data.

Consent to participate

Participants in the test clearly understood the risk and signed the relevant informed consent to participate in the experiment.

Consent for publication

The authors confirm that the publication of the work has been approved by all co-authors, if any, and that its publication has been approved (tacitly or explicitly) by the responsible authorities at the institution where the work is carried out.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The author confirms that the manuscript is not submitted to more than one journal for simultaneous consideration, that the submitted work is original and is not published elsewhere in any form or language, that the study is not split up into several parts to increase the quantity of submissions and submitted to various journals or to one journal over time, and that results are presented clearly, honestly, and without fabrication, falsification, or inappropriate data manipulation.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, H., Wu, D., Zheng, Y. et al. A rapid generation method of models in machining processes for real-time human–machine interaction with virtual-real fusion. Int J Adv Manuf Technol (2024). https://doi.org/10.1007/s00170-024-13736-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00170-024-13736-z

Keywords

Search

Navigation