Abstract
Numerical control (NC) codes verification is an important issue in computer numerical control (CNC) machining simulation because wrong NC codes will lead to the workpiece scrap and collision. The NC code verification methods both in physical space and cyber space (such as 3D computer graphics environment) have been widely investigated in recent years. However, physical verification methods have the problems that the simulation takes time and improper operations may cause danger. On the other hand, cyber verification methods only support some types of machines and cannot reflect the actual conditions of machine tools. This study proposes a cyber-physical prototype system for NC codes verification and CNC machining simulation. Based on the RGB-D camera, the depth-to-stereo model is constructed to obtain the 3D information in images. Without connecting with the CNC controller, the cutting tool and workpiece coordinate system (WCS) movement information in physical space can be got from images captured by the RGB-D camera through a convolutional neural network (CNN). Workpiece size and NC codes are imported into cyber space to render virtual workpiece with augmented reality (AR) technology. So that the operator can directly see the virtual workpiece in the physical machining scene. The virtual workpiece is machined by the cyber-physical system according to cutting tool movement in physical space. This research further confirms the feasibility of using computer vision (CV) methods to build the cyber-physical CNC simulation system based on an RGB-D camera. The potential application of the system is to obtain simulation results from CNC machine tools (especially those that are forbidden to connect the controller) and transfer the machining results to the Internet of Things (IoT).
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Acal, A. P., & Lobera, A. S. (2007). Virtual reality simulation applied to a numerical control milling machine. International Journal on Interactive Design and Manufacturing, 1(3), 143–154. https://doi.org/10.1007/s12008-007-0016-2
Adarsh, P., Rathi, P., & Kumar, M. (2020). YOLO v3-Tiny: Object detection and recognition using one stage improved model. In: 2020 6th international conference on advanced computing and communication systems, Coimbatore, India. https://doi.org/10.1109/ICACCS48705.2020.9074315.
Altintas, Y., Kersting, P., Biermann, D., Budak, E., Denkena, B., & Lazoglu, I. (2014). Virtual process systems for part machining operations. CIRP Annals, 63(2), 585–605. https://doi.org/10.1016/j.cirp.2014.05.007
Anderson, R. (1978). Detecting and eliminating collisions in NC machining. Computer-Aided Design, 10(4), 231–237. https://doi.org/10.1016/0010-4485(78)90058-1
Gatouillat, A., Badr, Y., Massot, B., & Sejdić, E. (2018). Internet of medical things: A review of recent contributions dealing with cyber-physical systems in medicine. IEEE Internet of Things Journal, 5(5), 3810–3822. https://doi.org/10.1109/JIOT.2018.2849014
Hsu, P.-L., & Yang, W.-T. (1993). Realtime 3D simulation of 3-axis milling using isometric projection. Computer-Aided Design, 25(4), 215–224. https://doi.org/10.1016/0010-4485(93)90052-P
Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, Lille, France. https://doi.org/10.48550/arXiv.1502.03167.
Kiswanto, G., & Ariansyah, D. (2013). Development of augmented reality (AR) for machining simulation of 3-axis CNC milling. In: 2013 international conference on advanced computer science and information systems, Sanur Bali, Indonesia. https://doi.org/10.1109/ICACSIS.2013.6761566.
Lee, S.-K., & Ko, S.-L. (2002). Development of simulation system for machining process using enhanced Z map model. Journal of Materials Processing Technology, 130, 608–617. https://doi.org/10.1016/S0924-0136(02)00761-6
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., et al. (2014). Microsoft coco: Common objects in context. European Conference on Computer Vision, Zurich, Switzerland. https://doi.org/10.48550/arXiv.1405.0312
Liu, C., Cao, S., Tse, W., & Xu, X. (2017). Augmented reality-assisted intelligent window for cyber-physical machine tools. Journal of Manufacturing Systems, 44, 280–286. https://doi.org/10.1016/j.jmsy.2017.04.008
Monostori, L., Kádár, B., Bauernhansl, T., Kondoh, S., Kumara, S., Reinhart, G., et al. (2016). Cyber-physical systems in manufacturing. CIRP Annals, 65(2), 621–641. https://doi.org/10.1016/j.cirp.2016.06.005
Ning, H., Ye, X., Bouras, M. A., Wei, D., & Daneshmand, M. (2018). General cyberspace: Cyberspace and cyber-enabled spaces. IEEE Internet of Things Journal, 5(3), 1843–1856. https://doi.org/10.1109/JIOT.2018.2815535
Ong, S.-K., Zhang, J., Shen, Y., & Nee, A. Y. (2011). Augmented reality in product development and manufacturing. In B. Furht (Ed.), Handbook of augmented reality (pp. 651–669). Springer.
Paelke, V. (2014). Augmented reality in the smart factory: Supporting workers in an industry 4.0. environment. In Proceedings of the 2014 IEEE emerging technology and factory automation, Barcelona, Spain. https://doi.org/10.1109/ETFA.2014.7005252.
Ragni, M., Perini, M., Setti, A., & Bosetti, P. (2018). ARTool Zero: Programming trajectory of touching probes using augmented reality. Computers & Industrial Engineering, 124, 462–473. https://doi.org/10.1016/j.cie.2018.07.026
Rajkumar, R., Lee, I., Sha, L., & Stankovic, J. (2010). Cyber-physical systems: the next computing revolution. In 47th Design automation conference, Anaheim, California, USA. https://doi.org/10.1145/1837274.1837461.
Redelinghuys, A., Basson, A. H., & Kruger, K. (2020). A six-layer architecture for the digital twin: A manufacturing case study implementation. Journal of Intelligent Manufacturing, 31, 1383–1402. https://doi.org/10.1007/s10845-019-01516-6
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA. https://doi.org/10.1109/CVPR.2016.91
Redmon, J., & Farhadi, A. (2018). Yolov3: An incremental improvement. https://doi.org/10.48550/arXiv.1804.02767.
Soori, M., Arezoo, B., & Habibi, M. (2013). Dimensional and geometrical errors of three-axis CNC milling machines in a virtual machining system. Computer-Aided Design, 45(11), 1306–1313. https://doi.org/10.1016/j.cad.2013.06.002
Tao, F., & Zhang, M. (2017). Digital twin shop-floor: A new shop-floor paradigm towards smart manufacturing. IEEE Access, 5, 20418–20427. https://doi.org/10.1109/ACCESS.2017.2756069
Tong, X., Liu, Q., Pi, S., & Xiao, Y. (2020). Real-time machining data application and service based on IMT digital twin. Journal of Intelligent Manufacturing, 31, 1113–1132. https://doi.org/10.1007/s10845-019-01500-0
Wang, X., Ong, S., & Nee, A.Y.-C. (2016). Multi-modal augmented-reality assembly guidance based on bare-hand interface. Advanced Engineering Informatics, 30(3), 406–421. https://doi.org/10.1016/j.aei.2016.05.004
Yang, L., Zhang, L., Dong, H., Alelaiwi, A., & El Saddik, A. (2015). Evaluating and improving the depth accuracy of Kinect for Windows v2. IEEE Sensors Journal, 15(8), 4275–4285. https://doi.org/10.1109/JSEN.2015.2416651
Younis, O., & Moayeri, N. (2017). Employing cyber-physical systems: Dynamic traffic light control at road intersections. IEEE Internet of Things Journal, 4(6), 2286–2296. https://doi.org/10.1109/JIOT.2017.2765243
Zhang, J., Ong, S.-K., & Nee, A. Y. (2010). A multi-regional computation scheme in an AR-assisted in situ CNC simulation environment. Computer-Aided Design, 42(12), 1167–1177. https://doi.org/10.1016/j.cad.2010.06.007
Zhang, J., Ong, S., & Nee, A. (2006). A volumetric model-based CNC simulation and monitoring system in augmented environments. In 2006 International conference on cyberworlds, Lausanne, Switzerland. https://doi.org/10.1109/CW.2006.11.
Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11), 1330–1334. https://doi.org/10.1109/34.888718
Zhao, Z., Wang, S., Wang, Z., Wang, S., & Yang, B. (2020). Surface roughness stabilization method based on digital twin-driven machining parameters self-adaption adjustment: A case study in five-axis machining. Journal of Intelligent Manufacturing, 33, 943–952. https://doi.org/10.1007/s10845-020-01698-4
Zhu, L., Li, H., Liang, W., & Wang, W. (2015). A web-based virtual CNC turn-milling system. The International Journal of Advanced Manufacturing Technology, 78(1), 99–113. https://doi.org/10.1007/s00170-014-6649-y
Acknowledgements
This study is funded by the Fundamental Research Funds for the Central Universities (Grant No. NT2021019), National Natural Science Foundation of China (Grant No. 51775279).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interest
The author declares that there is no conflict of interest.
Ethical Approval
Every ethical requirement has been met and observed.
Consent to Participate
Not applicable.
Consent to Publish
The editor has my (author) consent to publish.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wang, P., Yang, WA. & You, Y. A cyber-physical prototype system in augmented reality using RGB-D camera for CNC machining simulation. J Intell Manuf 34, 3637–3658 (2023). https://doi.org/10.1007/s10845-022-02021-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-022-02021-z