Skip to main content

Object Detection Using ESP32 Cameras for Quality Control of Steel Components in Manufacturing Structures

Arabian Journal for Science and Engineering (2022)Cite this article

Abstract

Automation in the construction industry has become more appealing in recent years. Although the industry fosters material mixing and structural application, automating quality control is not investigated broadly. The industry relies on manual inspection, leading to inaccuracy and lower productivity. This research focuses on the quality check of bolted steel members using automating ESP32 camera to detect missing bolts. Earlier studies focused on improving the quality of tightening the bolts. However, the major problem of missing bolts has not been extensively addressed. Inadequate bolting of steel members causes a considerable reduction in mechanical strength, and it may cause a structure to fail. Hence, this paper aims to detect missing bolts in assembling steel structures. The study was conducted by developing a system that utilizes an ESP32 camera module to capture the steel members in real-time. Captured video is processed in Visual Studio (C++ language), another approach carried out in the study is using a faster region-based convolutional neural network (Faster R-CNN) where it extracts the area of interest, the bolts, and holes in the steel members with the assistance of an image dataset and training. The trained model can be used to detect bolts and holes. The results showed that the developed system is reliable and can alter the user to any missing bolts, having TensorFlow object detection with the Faster R-CNN algorithm successfully provided desired results with 95% precision. This technique increases the efficiency of quality monitoring. Consequently, the steel manufacturing industry can rely on smart cameras to monitor the quality control of steel frames, leading to productive output. This also saves the workers time from performing the tedious task of inspecting every steel member and assists in maintaining the quality of the assembled steel structures.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25

References

  1. Martinez, P.: Automatic intelligent inspection systems for quality control: a case of defects in light-gauge steel frame assembly manufacturing, In: Department of Civil and Environmental Engineering. University of Alberta Alberta (2020)

  2. Kazemian, A., et al.: Computer vision for real-time extrusion quality monitoring and control in robotic construction. Autom. Constr. 101, 92–98 (2019)

    Article  Google Scholar 

  3. Shakor, P., et al.: Review of emerging additive manufacturing technologies in 3D printing of cementitious materials in the construction industry. Front. Built Environ. 4, 85 (2019)

    Article  Google Scholar 

  4. Martinez, P.; Al-Hussein, M.; Ahmad, R.: Intelligent vision-based online inspection system of screw-fastening operations in light-gauge steel frame manufacturing. Int. J. Adv. Manuf. Technol. 109(3), 645–657 (2020)

    Article  Google Scholar 

  5. Steel, H.L Standard Methods for Steel Building Structure Connection (2022). Available from: https://havitsteelstructure.com/steel-building-structure-connection/

  6. Kamaruddin, S.S.; Mohammad, M.F.; Mahbub, R.: Barriers and impact of mechanisation and automation in construction to achieve better quality products. Procedia-Soc. Behav. Sci. 222, 111–120 (2016)

    Article  Google Scholar 

  7. Dhanya, P.S.A.: A study of risk factors in construction automation and robotics. Int. J. Res. Mech. Eng. 6(1), 4–6 (2019)

    Google Scholar 

  8. Shakor, P., et al.: Dimensional accuracy, flowability, wettability, and porosity in inkjet 3DP for gypsum and cement mortar materials. Autom. Constr. 110, 102964 (2020)

    Article  Google Scholar 

  9. Chen, Q.; de García Soto, B.; Adey, B.T.: Construction automation: Research areas, industry concerns and suggestions for advancement. Autom. Constr. 94, 22–38 (2018)

    Article  Google Scholar 

  10. Irvine, D.N., Narayanan M.: Inspection and quality control-an overview. In: IEEE Technical Applications Conference and Workshops, Northcon (1995)

  11. Boukamp, F.; Akinci, B.: Automated processing of construction specifications to support inspection and quality control. Autom. Constr. 17(1), 90–106 (2007)

    Article  Google Scholar 

  12. Park, J.-H., et al.: Vision-based technique for bolt-loosening detection in wind turbine tower. Wind Struct. Int. J. 21, 709–726 (2015)

    Article  Google Scholar 

  13. Pham, H.C., et al.: Bolt-loosening monitoring framework using an image-based deep learning and graphical model. Sensors 20(12), 3382 (2020)

    Article  MathSciNet  Google Scholar 

  14. Yuan, C., et al.: Near real-time bolt-loosening detection using mask and region-based convolutional neural network. Struct. Control. Health Monit. 28(7), e2741 (2021)

    Article  Google Scholar 

  15. Shakor, P., Gowripalan, N.: Pressure Exerted on Formwork and Early Age Shrinkage of Self-Compacting Concrete. Concrete in Australia (2020)

  16. Gowripalan, N.; Shakor, P.; Rocker, P.: Pressure exerted on formwork by self-compacting concrete at early ages: a review. Case Stud. Constr. Mater. 15, e00642 (2021)

    Google Scholar 

  17. Riordan, A.D.O., et al.: Object recognition within smart manufacturing. Procedia Manuf. 38, 408–414 (2019)

    Article  Google Scholar 

  18. Huynh, T.-C., et al.: Quasi-autonomous bolt-loosening detection method using vision-based deep learning and image processing. Autom. Constr. 105, 1–18 (2019)

    Article  Google Scholar 

  19. Zhang, Y., et al.: Autonomous bolt loosening detection using deep learning. Struct. Health Monit. 19(1), 105–122 (2020)

    Article  Google Scholar 

  20. Ramana, L.; Choi, W.; Cha, Y.-J.: Fully automated vision-based loosened bolt detection using the Viola-Jones algorithm. Struct. Health Monit. 18(2), 422–434 (2018)

    Article  Google Scholar 

  21. Green, M.: 'Errors in Construction' Found at Failed Kennedy Bridge Joint, Records Show. WRDB.com (2021). Available from: https://www.wdrb.com/in-depth/errors-in-construction-found-at-failed-kennedy-bridge-joint-records-show/article_4ca2e518-eb17-11eb-8f91-6b1de83ed313.html

  22. Huynh, T.-C.: Vision-based autonomous bolt-looseness detection method for splice connections: design, lab-scale evaluation, and field application. Autom. Constr. 124, 1–22 (2021)

    Article  Google Scholar 

  23. Roboflow: Roboflow (2022). Available from: https://roboflow.com/

  24. Tutorialspoint: Google Colab – Introduction (2022). Available from: https://www.tutorialspoint.com/google_colab/google_colab_quick_guide.htm

  25. AI, T.: What is a Canny Edge Detection Algorithm? (2020). Available from: https://towardsai.net/p/computer-vision/what-is-a-canny-edge-detection-algorithm.

  26. Bear, J.H.: The HSV Color Model in Graphic Design. Lifewire (2020). Available from: https://www.lifewire.com/what-is-hsv-in-design-1078068

  27. Karim, R.: 10 Stochastic Gradient Descent Optimisation Algorithms + Cheatsheet. Towards Data Science (2018). Available from https://towardsdatascience.com/10-gradient-descent-optimisation-algorithms-86989510b5e9

  28. Jordan, J.: Setting the Learning Rate of Your Neural Network. Jeremyjordan.me (2018). Available from: https://www.jeremyjordan.me/nn-learning-rate/

  29. Sharma, S.: Epoch vs Batch Size vs Iterations. Towards Data Science (2017). Available from: https://towardsdatascience.com/epoch-vs-iterations-vs-batch-size-4dfb9c7ce9c9.

Download references

Author information

Authors and Affiliations

  1. Faculty of Engineering and IT, School of Civil and Environmental Engineering, University of Technology Sydney, Ultimo, NSW, 2007, Australia

    Sredha Vinod

  2. Remedial Engineer, Ironbridge Engineering, Surry Hills, NSW, 2010, Australia

    Sredha Vinod

  3. Technical College of Engineering, Sulaimani Polytechnic University, Sulaymaniyah, KRG, 46001, Iraq

    Pshtiwan Shakor

  4. Civil Engineering Department, College of Engineering, Tishk International University, Sulaymaniyah, KRG, 46001, Iraq

    Pshtiwan Shakor

  5. Institute of Construction Materials, Sydney, NSW, 2748, Australia

    Pshtiwan Shakor & Farid Sartipi

  6. Department of Civil and Environmental Engineering and Construction, University of Nevada, Las Vegas, NV, 89154, USA

    Moses Karakouzian

Authors
  1. Sredha Vinod
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pshtiwan Shakor
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Farid Sartipi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Moses Karakouzian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pshtiwan Shakor.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Vinod, S., Shakor, P., Sartipi, F. et al. Object Detection Using ESP32 Cameras for Quality Control of Steel Components in Manufacturing Structures. Arab J Sci Eng (2022). https://doi.org/10.1007/s13369-022-07562-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s13369-022-07562-2

Keywords

  • Object detection
  • ESP32 camera
  • Faster R-CNN
  • Quality control
  • Bolt and holes