Abstract
The construction industry is introducing autonomous heavy equipment to overcome labor shortages and improve productivity. For autonomous heavy equipment to work on earthmoving at sites, the equipment needs to recognize and understand ground surface types. However, the ground surface types are manually inspected in practice, and related studies are lacking. To address this issue, the authors developed and tested models that automatically classify ground surface types from images acquired by an unmanned aerial vehicle using a deep learning-based multi-label classification method that applies Binary Relevance (BR) and Label Powerset (LP) methods with Residual Neural Network (ResNet) and Vision Transformer classification network (VIT). The model performances were comparatively evaluated through experiments conducted on actual construction sites. The results showed that the BR model with ResNet is the best model in terms of automated ground surface type identification during earthmoving. The results are expected to broaden the understanding of complex and expansive construction sites for autonomous vehicles and thus facilitate deployment of autonomous heavy equipment by helping them to understand working areas and any obstacles on construction sites quickly and effectively, which will reduce the cost and time needed for on-site ground surface management.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abioye SO, Oyedele LO, Akanbi L, Ajayi A, Delgado JMD, Muhammad B, Olugbenga O, Ahmed A (2021) Artificial intelligence in the construction industry: A review of present status, opportunities and future challenges. Journal of Building Engineering 44:103299, DOI: https://doi.org/10.1016/j.jobe.2021.103299
Agustsson E, Timofte R (2017) Ntire 2017 challenge on single image super-resolution: Dataset and study. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops 126–135
Akinosho TD, Oyedele LO, Bilal M, Ajayi AO, Delgado MD, Akinade OO, Ahmed AA (2020) Deep learning in the construction industry: A review of present status and future innovations. Journal of Building Engineering 32:101827, DOI: https://doi.org/10.1016/j.jobe.2020.101827
Arashpour M, Ngo T, Li H (2021) Scene understanding in construction and buildings using image processing methods: A comprehensive review and a case study. Journal of Building Engineering 33: 101672, DOI: https://doi.org/10.1016/j.jobe.2020.101672
Associated General Contractors (AGC) (2020) 2020 Construction Outlook Survey Results National Results, https://tinyurl.com/3u8ew4d9
Azar ER, Kamat VR (2017) Earthmoving equipment automation: A review of technical advances and future outlook. Journal of Information Technology in Construction (ITcon) 22(13):247–265
Bang S, Baek F, Park S, Kim W, Kim H (2020) Image augmentation to improve construction resource detection using generative adversarial networks, cut-and-paste, and image transformation techniques. Automation in Construction 115:103198, DOI: https://doi.org/10.1016/j.autcon.2020.103198
Bang S, Hong Y, Kim H (2021) Proactive proximity monitoring with instance segmentation and unmanned aerial vehicle-acquired video-frame prediction. Computer-Aided Civil and Infrastructure Engineering 36(6):800–816, DOI: https://doi.org/10.1111/mice.12672
Bang S, Kim H, Kim H (2017b) Vision-based 2D map generation for monitoring construction sites using UAV Videos. In ISARC. Proceedings of the International Symposium on Automation and Robotics in Construction (Vol. 34). IAARC Publications, DOI: https://doi.org/10.22260/isarc2017/0116
Brucker Juricic B, Galic M, Marenjak S (2021) Review of the construction labour demand and shortages in the EU. Buildings 11(1):17, DOI:https://doi.org/10.3390/buildings11010017
Chen J, Huang H, Cohn AG, Zhou M, Zhang D, Man J (2022) A hierarchical DCNN-based approach for classifying imbalanced water inflow in rock tunnel faces. Tunnelling and Underground Space Technology 122:104399, DOI: https://doi.org/10.1016/j.tust.2022.104399
Chen J, Yang T, Zhang D, Huang H, Tian Y (2021) Deep learning based classification of rock structure of tunnel face. Geoscience Frontiers 12(1):395–404, DOI: https://doi.org/10.1016/j.gsf.2020.04.003
Cho S, Kim D, Kang J, Kim J, Hwang S, Kim S (2021) Smart construction global insight Vol.1. Center for Smart Construction Technology, http://www.smartcons1ruction.kr/library/report/view?idx=311
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929, DOI:https://doi.org/10.48550/arXiv.2010.11929
Fang W, Ding L, Zhong B, Love PE, Luo H (2018) Automated detection of workers and heavy equipment on construction sites: A convolutional neural network approach. Advanced Engineering Informatics 37:139–149, DOI: https://doi.org/10.1016/j.aei.2018.05.003
Guo Y, Xu Y, Li S (2020) Dense construction vehicle detection based on orientation-aware feature fusion convolutional neural network. Automation in Construction 112:103124, DOI: https://doi.org/10.1016/j.autcon.2020.103124
Ham Y, Han KK, Lin JJ, Golparvar-Fard M (2016) Visual monitoring of civil infrastructure systems via camera-equipped Unmanned Aerial Vehicles (UAVs): A review of related works. Visualization in Engineering 4(1):1–8, DOI: https://doi.org/10.1186/s40327-015-0029-z
Han S, Jiang Y, Bai Y (2022) Fast-PGMED: Fast and dense elevation determination for earthwork using drone and deep learning. Journal of Construction Engineering and Management 148(4):04022008, DOI: https://doi.org/10.1061/(ASCE)CO.1943-7862.0002256
Han J, Zhang D, Cheng G, Guo L, Ren J (2015) Object detection in optical remote sensing images based on weakly supervised learning and high-level feature learning. IEEE Transactions on Geoscience and Remote Sensing 53(6):3325–3337, DOI: https://doi.org/10.1109/TGRS.2014.2374218
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778
Huang H, Zhang C, Hammad A (2021) Effective scanning range estimation for using TLS in construction projects. Journal of Construction Engineering and Management 147(9):04021106, DOI: https://doi.org/10.1061/(ASCE)CO.1943-7862.0002127
Jiang Y, Bai Y (2020) Estimation of construction site elevations using drone-based orthoimagery and deep learning. Journal of Construction Engineering and Management 146(8), DOI: https://doi.org/10.1061/(ASCE)CO.1943-7862.0001869
Jiang Y, Bai Y (2021) Low-high orthoimage pairs-based 3d reconstruction for elevation determination using drone. Journal of Construction Engineering and Management 147(9), DOI: https://doi.org/10.1061/(ASCE)CO.1943-7862.0002067
Jiang Y, Bai Y, Han S (2020b) Determining ground elevations covered by vegetation on construction sites using drone-based orthoimage and convolutional neural network. Journal of Computing in Civil Engineering 34(6):04020049, DOI: https://doi.org/10.1061/(ASCE)CP.1943-5487.0000930
Jiang W, Zhou Y, Ding L, Zhou C, Ning X (2020a) UAV-based 3D reconstruction for hoist site mapping and layout planning in petrochemical construction. Automation in Construction 113:103137, DOI: https://doi.org/10.1016/j.autcon.2020.103137
Kim D, Lee S, Kamat VR (2020) Proximity prediction of mobile objects to prevent contact-driven accidents in co-robotic construction. Journal of Computing in Civil Engineering 34(4):04020022, DOI:https://doi.org/10.1061/(ASCE)CP.1943-5487.0000899
Kim DY (2014) A basic study on investigation of current craftmen status in korea construction site. Architectural Institute of Korea 30(11):81–88, DOI: https://doi.org/10.5659/JAIKSC.2014.30.11.81
Kim J, Chi S, Hwang B-G (2017) Vision-based activity analysis framework considering interactive operation of construction equipment. ASCE International Workshop on Computing in Civil Engineering 2017. American Society of Civil Engineers, Reston, VA: 162–170, DOI:https://doi.org/10.1061/9780784480830.021
Kim J, Chi S, Kwon T (2016) Construction entities tracking based on functional integration and online learning with site-customized datasets. Proceedings of the CIB World Building Congress 2016, Tampere, Finland: 1118–1128
Kim J, Ham Y, Chung Y, Chi S (2019a) Systematic camera placement framework for operation-level visual monitoring on construction jobsites. Journal of Construction Engineering and Management 145(4):04019019, DOI: https://doi.org/10.1061/(ASCE)CO.1943-7862.0001636
Kim K, Kim S, Shchur D (2021a) A UAS-based work zone safety monitoring system by integrating an internal traffic control plan (ITCP) and automated object detection in game engine environment. Automation in Construction 128:103736, DOI: https://doi.org/10.1016/j.autcon.2021.103736
Kim J, Lee S, Seo J, Lee DE, Choi HS (2021b) The integration of earthwork design review and planning using UAV-based point cloud and BIM. Applied Sciences 11(8):3435, DOI: https://doi.org/10.3390/app11083435
Kim D, Liu M, Lee S, Kamat VR (2019b) Remote proximity monitoring between mobile construction resources using camera-mounted UAVs. Automation in Construction 99:168–182, DOI: https://doi.org/10.1016/j.autcon.2018.12.014
Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980
Kussul N, Lavreniuk M, Skakun S, Shelestov A (2017) Deep learning classification of land cover and crop types using remote sensing data. IEEE Geoscience and Remote Sensing Letters 14(5):778–782, DOI: https://doi.org/10.1109/LGRS.2017.2681128
Leonida C (2020) Giving operators a helping hand. Engineering and Mining Journal 221(2):44–46
Liang J, Cao J, Sun G, Zhang K, Gool L van, Timofte R (2021) SwinIR: Image Restoration Using Swin Transformer 2021, https://github.com/JingyunLiang/SwinIR
Maggiori E, Tarabalka Y, Charpiat G, Alliez P (2016) Fully convolutional neural networks for remote sensing image classification. In Proceedings of the 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS) 5071–5074, New York: IEEE, DOI: https://doi.org/10.1109/IGARSS.2016.7730322
Matsui Y, Ito K, Aramaki Y, Fujimoto A, Ogawa T, Yamasaki T, Kiyoharu A (2017) Sketch-based manga retrieval using manga109 dataset. Multimed Tools and Applications 76(2017):21811–21838, DOI: https://doi.org/10.1007/s11042-016-4020-z
Ministry of Land, Infrastructure and Transport (MOLIT) (2016a) Road Construction Standard Specification
Ministry of Land, Infrastructure and Transport (MOLIT) (2016b) Road Design Standards
Mishra M, Lourenço PB, Ramana GV (2022) Structural health monitoring of civil engineering structures by using the internet of things: A review. Journal of Building Engineering 48:103954, DOI: https://doi.org/10.1016/j.jobe.2021.103954
Moon D, Chung S, Kwon S, Seo J, Shin J (2019) Comparison and utilization of point cloud generated from photogrammetry and laser scanning: 3D world model for smart heavy equipment planning. Automation in Construction 98:322–331, DOI: https://doi.org/10.1016/j.autcon.2018.07.020
Nguyen HA, Ha QP (2022) Robotic autonomous systems for earthmoving equipment operating in volatile conditions and teaming capacity: A survey. Robotica, 1–25, DOI: https://doi.org/10.1017/S0263574722000339
Pi Y, Nath ND, Behzadan AH (2020) Convolutional neural networks for object detection in aerial imagery for disaster response and recovery. Advanced Engineering Informatics 43:101009, DOI: https://doi.org/10.1016/j.aei.2019.101009
Qian Y, Leng J (2021) CIM-based modeling and simulating technology roadmap for maintaining and managing Chinese rural traditional residential dwellings. Journal of Building Engineering 44:103248, DOI: https://doi.org/10.1016/j.jobe.2021.103248
Read J, Pfahringer B, Holmes G, Frank E (2011) Classifier chains for multi-label classification. Machine Learning 85(3):333–359, DOI:https://doi.org/10.1007/978-3-642-04174-717
Seo J, Lee S, Kim J, Kim SK (2011) Task planner design for an automated excavation system. Automation in Construction 20(7):954–966, DOI: https://doi.org/10.1016/j.autcon.2011.03.013
Sharma O (2019) A new activation function for deep neural network. In 2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon), 84–86, DOI: https://doi.org/10.1109/COMITCon.2019.8862253
Sorower MS (2010) A literature survey on algorithms for multi-label learning. Oregon State University, Corvallis 18:1–25
Spolaôr N, Cherman EA, Monard MC, Lee HD (2013) A comparison of multi-label feature selection methods using the problem transformation approach. Electronic Notes in Theoretical Computer Science 292:135–151, DOI: https://doi.org/10.1016/j.entcs.2013.02.010
Tajeen H, Zhu Z (2014) Image dataset development for measuring construction equipment recognition performance. Automation in Construction 48: 1–10, DOI: https://doi.org/10.1016/j.autcon.2014.07.006
Timofte Eirikur R, Agustsson E, Van Gool L, Yang M-H, Zhang L (2017) NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results, 2017, http://www.vision.ee.ethz.ch/ntire17/
Torres HN, Ruiz JM, Chang GK, Anderson JL, Garber SI (2018) Automation in highway construction part I: Implementation challenges at state transportation departments and success stories (No. FHWA-HRT-16-030). United States. Federal Highway Administration. Office of Infrastructure Research and Development, https://rosap.ntl.bts.gov/view/dot/41947
Trimble Inc. (2022) Trimble WorksOS Datasheet, https://tinyurl.com/36n627xx
Tsoumakas G, Katakis I, Vlahavas I (2009) Mining multi-label data. In Data mining and knowledge discovery handbook, 667–685. Springer, Boston, MA, DOI: https://doi.org/10.1007/978-0-387-09823-434
U.S. Department of Transportation Federal Highway Administration (USDOT) (2014) Standard specifications for construction of roads and bridges on federal highway projects (FP-14), https://dotconstructioninc.com/wp-content/uploads/2018/12/FP14_Eng.pdf
United States Federal Highway Administration (FWHA) (2021) Determination of Improved Pavement Smoothness When Using 3D Modeling and Automatic Machine Guidance, https://tinyurl.com/2kbtucj2
Wortsman M, Ilharco G, Gadre SY, Roelofs R, Gontijo-Lopes R, Morcos AS, Namkoong H, Farhadi A, Carmon Y, Kornblith S, Schmidt L (2022) Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv preprint arXiv:2203.05482, https://proceedings.mlr.press/v162/wortsman22a.html
Xiao Y, Kamat VR, Lee S (2018) Monitoring excavation slope stability using drones. In Construction Research Congress 2018, 169–179, https://doi.org/10.1061/9780784481264.017
Xu M, Nie X, Li H, Cheng JC, Mei Z (2022) Smart construction sites: A promising approach to improving on-site HSE management performance. Journal of Building Engineering 49:104007, DOI:https://doi.org/10.1016/j.jobe.2022.104007
Xuehui A, Li Z, Zuguang L, Chengzhi W, Pengfei L, Zhiwei L (2021) Dataset and benchmark for detecting moving objects in construction sites. Automation in Construction 122:103482, DOI: https://doi.org/10.1016/j.autcon.2020.103482
Yamamoto H, Moteki M, Shao H, Ootuki T, Kanazawa H, Tanaka Y (2009) Basic technology toward autonomous hydraulic excavator. In 26th International Symposium on Automation and Robotics in Construction (ISARC 2009), 288–295
Zhang ML, Li YK, Liu XY, Geng X (2018) Binary relevance for multilabel learning: An overview. Frontiers of Computer Science 12(2):191–202, DOI: https://doi.org/10.1007/s11704-017-7031-7
Zhang S, Bogus SM, Lippitt CD, Kamat V, Lee S (2022) Implementing remote-sensing methodologies for construction research: An unoccupied airborne system perspective. Journal of Construction Engineering and Management 148(9):03122005, DOI: https://doi.org/10.1061/(ASCE)CO.1943-7862.0002347
Acknowledgments
This work was supported by the National R&D Project for Smart Construction Technology (23SMIP-A158708-04) funded by the Korea Agency of Infrastructure Technology Advancement under the Ministry of Land, Infrastructure, and Transport. This work was also supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00241758).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Won, D., Chi, S. & Choi, J.O. UAV Imagery-based Automatic Classification of Ground Surface Types for Earthworks. KSCE J Civ Eng (2024). https://doi.org/10.1007/s12205-024-1643-x
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s12205-024-1643-x