Abstract
This research focuses on developing a robot digital twin (DT) and the communication methods to connect it with the corresponding physical robot in collaborative human–robot construction work. Robots are being increasingly deployed on construction sites to assist human workers with physically demanding work tasks. Robot simulations in a process-level DT can be used to extend design models, such as building information modeling, to the construction phase for real-time monitoring of robot motion planning and control. Robots can be enabled to plan work tasks and execute them in the DT simulations. Once simulated tasks and trajectories are approved by human workers, commands can be sent to the physical robots to perform the tasks. However, a system to bridge a virtual DT and a physical robot and allow for such communication to occur is a capability that has not been readily available thus far, primarily due to the complexity involved in physical robot operations. This paper discusses the development of a system to bridge robot simulations and physical robots in construction and digital fabrication. The Gazebo robot simulator is used for DT, and the robot operating system is leveraged as the primary framework for bi-directional communication with the physical robots. The virtual robots in Gazebo receive planned trajectories from motion planners and then send the commands to the physical robots for execution. Two different robot control modes, i.e., joint angle control mode and Cartesian path control mode, are developed to accommodate various construction strategies. The system is implemented in a digital fabrication case study with a full-scale KUKA KR120 six-degrees-of-freedom robotic arm mounted on a track system. We evaluated the system by comparing the data transmission time, joint angles, and end-effector pose between the virtual and physical robot using several planned trajectories and calculated the average and maximum mean square errors. The results showed that the proposed real-time process-level robot DT system can plan the robot trajectory inside the virtual environment and execute it in the physical environment with high accuracy and real-time performance, offering the opportunity for further development and deployment of the collaborative human–robot work paradigm on real construction sites.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aheleroff S, Polzer J, Huang H, Zhu Z, Tomzik D, Lu Y, Lin Y, Xu X (2020) Smart manufacturing based on digital twin technologies. In: Machado C, Davim JP (eds) Industry 4.0: challenges, trends, and solutions in management and engineering. CRC Press, Boca Raton, p 77. https://doi.org/10.1201/9781351132992-3
Al-Sehrawy R, Kumar B (2020) Digital twins in architecture, engineering, construction and operations. a brief review and analysis. In: Toledo Santos E, Scheer S (eds) Proceedings of the international conference on computing in civil and building engineering (ICCCBE). Springer International Publishing, São Paulo, Brazil (Online), 2020, pp 924–939. https://doi.org/10.1007/978-3-030-51295-8_64
Beckhoff, TwinCAT ADS, Beckhoff (2021) https://github.com/Beckhoff/ADS. Accessed 6 Feb 2021
Bilberg A, Malik AA (2019) Digital twin driven human–robot collaborative assembly. CIRP Ann 68:499–502. https://doi.org/10.1016/j.cirp.2019.04.011
Bosché F, Ahmed M, Turkan Y, Haas CT, Haas R (2015) The value of integrating Scan-to-BIM and Scan-vs-BIM techniques for construction monitoring using laser scanning and BIM: the case of cylindrical MEP components. Autom Constr 49:201–213. https://doi.org/10.1016/j.autcon.2014.05.014
Bruckmann T, Mattern H, Spenglerc A, Reichert C, Malkwitz A, König M (2016) Automated construction of masonry buildings using cable-driven parallel robots. In: Proceedings of the international symposium on automation and robotics in construction (ISARC), IAARC, Auburn, AL, USA, 2016, pp 332–340. https://doi.org/10.22260/ISARC2016/0041
Cai Y, Wang Y, Burnett M (2020) Using augmented reality to build digital twin for reconfigurable additive manufacturing system. J Manuf Syst. https://doi.org/10.1016/j.jmsy.2020.04.005 (in press)
Coleman DT, Sucan IA, Chitta S, Correll N (2014) Reducing the barrier to entry of complex robotic software: a MoveIt! case study. J Softw Eng Robot 5:3–16. https://doi.org/10.6092/JOSER_2014_05_01_p3
Colledani M, Terkaj W, Tolio T (2009) Product-process-system information formalization. In: Tolio T (ed) Design of flexible production systems: methodologies and tools. Springer, Berlin, pp 63–86. https://doi.org/10.1007/978-3-540-85414-2_4
Delbrügger T, Lenz LT, Losch D, Roßmann J (2017) A navigation framework for digital twins of factories based on building information modeling. In: Proceedings of the IEEE international conference on emerging technologies and factory automation (ETFA), IEEE, Limassol, Cyprus, 2017, pp 1–4. https://doi.org/10.1109/ETFA.2017.8247712
Dimitrov A, Golparvar-Fard M (2015) Segmentation of building point cloud models including detailed architectural/structural features and MEP systems. Autom Constr 51:32–45. https://doi.org/10.1016/j.autcon.2014.12.015
Eadie R, Browne M, Odeyinka H, McKeown C, McNiff S (2013) BIM implementation throughout the UK construction project lifecycle: An analysis. Autom Constr 36:145–151. https://doi.org/10.1016/j.autcon.2013.09.001
Eversmann P, Gramazio F, Kohler M (2017) Robotic prefabrication of timber structures: towards automated large-scale spatial assembly. Constr Robot 1:49–60. https://doi.org/10.1007/s41693-017-0006-2
Farrell RG, Lenchner J, Kephjart JO, Webb AM, Muller MJ, Erikson TD, Melville DO, Bellamy RKE, Gruen DM, Connell JH, Soroker D, Aaron A, Trewin SM, Ashoori M, Ellis JB, Gaucher BP, Gil D (2016) Symbiotic cognitive computing. AI Mag 37:81–93. https://doi.org/10.1609/aimag.v37i3.2628
Feng C, Xiao Y, Willette A, McGee W, Kamat VR (2015) Vision guided autonomous robotic assembly and as-built scanning on unstructured construction sites. Autom Constr 59:128–138. https://doi.org/10.1016/j.autcon.2015.06.002
Freedy A, DeVisser E, Weltman G, Coeyman N (2007) Measurement of trust in human-robot collaboration. In: Proceedings of the international symposium on collaborative technologies and systems, IEEE, Orlando, FL, USA, 2007, pp 106–114. https://doi.org/10.1109/CTS.2007.4621745
Hamledari H, McCabe B, Davari S, Shahi A (2017) Automated schedule and progress updating of IFC-based 4D BIMs. J Comput Civ Eng 31:04017012. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000660
Hamledari H, Rezazadeh Azar E, McCabe B (2018) IFC-based development of as-built and as-is BIMs using construction and facility inspection data: site-to-BIM data transfer automation. J Comput Civ Eng 32:04017075. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000727
Hentout A, Aouache M, Maoudj A, Akli I (2019) Human–robot interaction in industrial collaborative robotics: a literature review of the decade 2008–2017. Adv Robot 33:764–799. https://doi.org/10.1080/01691864.2019.1636714
Kam HR, Lee S-H, Park T, Kim C-H (2015) RViz: a toolkit for real domain data visualization. Telecommun Syst 60:337–345. https://doi.org/10.1007/s11235-015-0034-5
Kamat VR, Martinez JC (2005) Large-scale dynamic terrain in three-dimensional construction process visualizations. J Comput Civ Eng 19:160–171. https://doi.org/10.1061/(ASCE)0887-3801(2005)19:2(160)
Kan C, Anumba CJ (2019) Digital twins as the next phase of cyber-physical systems in construction. In: Proceedings of the ASCE international conference on computing in civil engineering (I3CE), ASCE, Atlanta, Georgia, 2019, pp 256–264. https://doi.org/10.1061/9780784482438.033
Koenig N, Howard A (2004) Design and use paradigms for Gazebo, an open-source multi-robot simulator. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE, Sendai, Japan, 2004, pp 2149–2154. https://doi.org/10.1109/IROS.2004.1389727
Kopetz H (2011) Real-time systems: design principles for distributed embedded applications. Springer, New York
Liang C-J, Lundeen KM, McGee W, Menassa CC, Lee S, Kamat VR (2018) Stacked hourglass networks for marker-less pose estimation of articulated construction robots. In: Proceedings of the international symposium on automation and robotics in construction (ISARC), IAARC, Berlin, Germany, 2018, pp 859–865. https://doi.org/10.22260/ISARC2018/0120
Liang C-J, Lundeen KM, McGee W, Menassa CC, Lee S, Kamat VR (2019a) Fast dataset collection approach for articulated equipment pose estimation. In: Proceedings of the ASCE international conference on computing in civil engineering (I3CE), ASCE, Atlanta, GA, USA, 2019a, pp 146–152. https://doi.org/10.1061/9780784482438.019
Liang C-J, Lundeen KM, McGee W, Menassa CC, Lee S, Kamat VR (2019b) A vision-based marker-less pose estimation system for articulated construction robots. Autom Constr 104:80–94. https://doi.org/10.1016/j.autcon.2019.04.004
Liang C-J, Kamat VR, Menassa CC (2020) Teaching robots to perform quasi-repetitive construction tasks through human demonstration. Autom Constr 120:103370. https://doi.org/10.1016/j.autcon.2020.103370
Liang C-J, Wang X, Kamat VR, Menassa CC (2021) Human–robot collaboration in construction: classification and research trends. J Constr Eng Manag 147:03121006. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002154
Liang C-J, Kamat VR, Menassa CC, McGee W (2022) Trajectory-based skill learning for overhead construction robots using generalized cylinders with orientation. J Comput Civ Eng 36:04021036. https://doi.org/10.1061/(ASCE)CP.1943-5487.0001004
Light RA (2017) Mosquitto: server and client implementation of the MQTT protocol. J Open Source Softw 2:265. https://doi.org/10.21105/joss.00265
Lin JJ, Lee JY, Golparvar-Fard M (2019) Exploring the potential of image-based 3D geometry and appearance reasoning for automated construction progress monitoring. In: Proceedings of the ASCE international conference on computing in civil engineering (I3CE), ASCE, Atlanta, GA, USA, 2019, pp 162–170. https://doi.org/10.1061/9780784482438.021
Linner T, Shrikathiresan A, Vetrenko M, Ellmann B, Bock T (2011) Modeling and operating robotic environments using Gazebo/ROS. In: Proceedings of the international symposium on automation and robotics in construction (ISARC), IAARC, Seoul, Korea, 2011, pp 957–962. https://doi.org/10.22260/ISARC2011/0177
Lu Y, Xu X (2018) Resource virtualization: a core technology for developing cyber-physical production systems. J Manuf Syst 47:128–140. https://doi.org/10.1016/j.jmsy.2018.05.003
Lu Q, Xie X, Parlikad AK, Schooling JM, Konstantinou E (2020a) Moving from building information models to digital twins for operation and maintenance. In: Proceedings of the Institution of Civil Engineers—smart infrastructure and construction (2020a), pp 1–11. https://doi.org/10.1680/jsmic.19.00011
Lu Q, Xie X, Parlikad AK, Schooling JM (2020b) Digital twin-enabled anomaly detection for built asset monitoring in operation and maintenance. Autom Constr 118:103277. https://doi.org/10.1016/j.autcon.2020.103277
Lundeen KM, Kamat VR, Menassa CC, McGee W (2017) Scene understanding for adaptive manipulation in robotized construction work. Autom Constr 82:16–30. https://doi.org/10.1016/j.autcon.2017.06.022
Lundeen KM, Kamat VR, Menassa CC, McGee W (2019) Autonomous motion planning and task execution in geometrically adaptive robotized construction work. Autom Constr 100:24–45. https://doi.org/10.1016/j.autcon.2018.12.020
Macher H, Landes T, Grussenmeyer P (2017) From point clouds to building information models: 3D semi-automatic reconstruction of indoors of existing buildings. Appl Sci 7:1030. https://doi.org/10.3390/app7101030
Madni AM, Madni CC, Lucero SD (2019) Leveraging digital twin technology in model-based systems engineering. Systems 7:7. https://doi.org/10.3390/systems7010007
Malik AA, Brem A (2021) Digital twins for collaborative robots: a case study in human-robot interaction. Robot Comput Integr Manuf 68:102092. https://doi.org/10.1016/j.rcim.2020.102092
Marshall MQ, Redovian C (2019) An application of a digital twin to robotic system design for an unstructured environment. In: Proceedings of the ASME international mechanical engineering congress and exposition, ASME, Salt Lake City, UT, USA, 2019, p V02BT02A010. https://doi.org/10.1115/IMECE2019-11337
Mertens J, Challenger M, Vanherpen K, Denil J (2020) Towards real-time cyber-physical systems instrumentation for creating digital twins. In: Proceedings of the spring simulation conference (SpringSim), IEEE, Fairfax, VA, USA, 2020, pp 1–12. https://doi.org/10.22360/SpringSim.2020.CPS.001
Mohammed A, Schmidt B, Wang L (2017) Active collision avoidance for human–robot collaboration driven by vision sensors. Int J Comput Integr Manuf 30:970–980. https://doi.org/10.1080/0951192X.2016.1268269
Musić S, Hirche S (2017) Control sharing in human-robot team interaction. Annu Rev Control 44:342–354. https://doi.org/10.1016/j.arcontrol.2017.09.017
Naboni R, Kunic A (2019) A computational framework for the design and robotic manufacturing of complex wood structures. In: Proceedings of the education and research in computer aided architectural design in Europe and Iberoamerican Society of Digital Graphics, joint conference, Porto, Portugal, 2019, pp 189–196. https://doi.org/10.5151/proceedings-ecaadesigradi2019_488
Nikolakis N, Maratos V, Makris S (2019) A cyber physical system (CPS) approach for safe human–robot collaboration in a shared workplace. Robot Comput Integr Manuf 56:233–243. https://doi.org/10.1016/j.rcim.2018.10.003
Ochmann S, Vock R, Wessel R, Klein R (2016) Automatic reconstruction of parametric building models from indoor point clouds. Comput Graph 54:94–103. https://doi.org/10.1016/j.cag.2015.07.008
OCTOPUZ (2021) Robot programming and simulation software: offline robotic program (OLRP). https://octopuz.com/. Accessed 7 Dec 2021
Quigley M, Gerkey B, Conley K, Faust J, Foote T, Leibs J, Berger E, Wheeler R, Ng AY (2009) ROS: an open-source robot operating system. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), IEEE, Kobe, Japan, 2009, p 5. https://www.semanticscholar.org/paper/ROS%3A-an-open-source-Robot-Operating-System-Quigley/d45eaee8b2e047306329e5dbfc954e6dd318ca1e#citing-papers. Accessed 5 Jan 2021.
RoboDK (2021) Simulator for industrial robots and offline programming. https://robodk.com/index. Accessed 7 Dec 2021
Sampaio AZ, Berdeja E (2017) Collaborative BIM environment as a support to conflict analysis in building design. In: Proceedings of the experiment@international conference (Exp.at’17), IEEE, Faro, Portugal, 2017, pp 77–82. https://doi.org/10.1109/EXPAT.2017.7984348
Sartori A, Schlette C (2021) Visual programming of a human-machine interface for a multi-robot support system. In: Proceedings of the IEEE international conference on industrial cyber-physical systems (ICPS), IEEE, Victoria, Canada, 2021, pp 387–392. https://doi.org/10.1109/ICPS49255.2021.9468200
Schmidtler J, Knott V, Hölzel C, Bengler K (2015) Human centered assistance applications for the working environment of the future. Occup Ergon 12:83–95. https://doi.org/10.3233/OER-150226
Shahmiri F, Ficca J (2016) A model for real-time control of industrial robots. In: Proceedings of the international symposium on automation and robotics in construction (ISARC), IAARC, Auburn, AL, USA, 2016, pp 1065–1072. https://doi.org/10.22260/ISARC2016/0128
Sharif S, Gentry TR, Sweet LM (2016) Human–robot collaboration for creative and integrated design and fabrication processes. In: Proceedings of the international symposium on automation and robotics in construction (ISARC), IAARC, Auburn, AL, USA, 2016, pp 596–604. https://doi.org/10.22260/ISARC2016/0072
Shin KG, Ramanathan P (1994) Real-time computing: a new discipline of computer science and engineering. Proc IEEE 82:6–24. https://doi.org/10.1109/5.259423
Söderberg R, Wärmefjord K, Carlson JS, Lindkvist L (2017) Toward a digital twin for real-time geometry assurance in individualized production. CIRP Ann 66:137–140. https://doi.org/10.1016/j.cirp.2017.04.038
Song L, Eldin NN (2012) Adaptive real-time tracking and simulation of heavy construction operations for look-ahead scheduling. Autom Constr 27:32–39. https://doi.org/10.1016/j.autcon.2012.05.007
Stojanovic V, Trapp M, Richter R, Hagedorn B, Döllner J (2018) Towards the generation of digital twins for facility management based on 3D point clouds. In: Proceedings of the ARCOM 34th Annual Conference, Belfast, UK, 2018, pp 270–279. https://www.researchgate.net/publication/325737190_Towards_The_Generation_of_Digital_Twins_for_Facility_Management_Based_on_3D_Point_Clouds. Accessed 31 Mar 2021
Tabar RS, Wärmefjord K, Söderberg R, Lindkvist L (2020) Efficient spot welding sequence optimization in a geometry assurance digital twin. J Mech Des 142:1–8. https://doi.org/10.1115/1.4046436
Tandur S (2015) Towards a new BIM “dimension”—translating BIM data into actual construction using robotics. In: Proceedings of the international symposium on automation and robotics in construction (ISARC), IAARC, Oulu, Finland, 2015, pp 1–7. https://doi.org/10.22260/ISARC2015/0051
Usmanov V, Bruzl M, Svoboda P, Šulc R (2017) Modelling of industrial robotic brick system. In: Proceedings of the international symposium on automation and robotics in construction (ISARC), IAARC, Taipei, Taiwan, 2017, pp 1013–1020. https://doi.org/10.22260/ISARC2017/0140
Vasey L, Felbrich B, Prado M, Tahanzadeh B, Menges A (2020) Physically distributed multi-robot coordination and collaboration in construction. Constr Robot 4:3–18. https://doi.org/10.1007/s41693-020-00031-y
Wang C, Cho YK (2015) Smart scanning and near real-time 3D surface modeling of dynamic construction equipment from a point cloud. Autom Constr 49:239–249. https://doi.org/10.1016/j.autcon.2014.06.003
Wang L, Gao R, Váncza J, Krüger J, Wang XV, Makris S, Chryssolouris G (2019) Symbiotic human-robot collaborative assembly. CIRP Ann 68:701–726. https://doi.org/10.1016/j.cirp.2019.05.002
Wang X, Liang C-J, Menassa CC, Kamat VR (2021) Interactive and immersive process-level digital twin for collaborative human–robot construction work. J Comput Civ Eng 35:04021023. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000988
Wu T-H, Wu F, Liang C-J, Li Y-F, Tseng C-M, Kang S-C (2017) A virtual reality tool for training in global engineering collaboration. Univ Access Inf Soc 18:243–255. https://doi.org/10.1007/s10209-017-0594-0
Xiao Y, Taguchi Y, Kamat VR (2018) Coupling point cloud completion and surface connectivity relation inference for 3D modeling of indoor building environments. J Comput Civ Eng 32:04018033. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000776
Xu L, Feng C, Kamat VR, Menassa CC (2019) An occupancy grid mapping enhanced visual slam for real-time locating applications in indoor gps-denied environments. Autom Constr 104:230–245. https://doi.org/10.1016/j.autcon.2019.04.011
Xue F, Lu W, Chen K, Zetkulic A (2019) From semantic segmentation to semantic registration: derivative-free optimization–based approach for automatic generation of semantically rich as-built building information models from 3D point clouds. J Comput Civ Eng 33:04019024. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000839
Yang C-H, Wu T-H, Xiao B, Kang S-C (2019) Design of a robotic software package for modular home builder. In: Proceedings of the international symposium on automation and robotics in construction (ISARC), IAARC, Banff, AB, Canada, 2019, pp 1217–1222. https://doi.org/10.22260/ISARC2019/0163
Zhuang C, Liu J, Xiong H (2018) Digital twin-based smart production management and control framework for the complex product assembly shop-floor. Int J Adv Manuf Technol 96:1149–1163. https://doi.org/10.1007/s00170-018-1617-6
Funding
The work presented in this paper was supported financially by United States National Science Foundation Awards (#2025805 and #2128623). Any opinions, findings, and conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the United States National Science Foundation.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Liang, CJ., McGee, W., Menassa, C.C. et al. Real-time state synchronization between physical construction robots and process-level digital twins. Constr Robot 6, 57–73 (2022). https://doi.org/10.1007/s41693-022-00068-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41693-022-00068-1