Skip to main content

Advertisement

SpringerLink
Log in

An intelligent hexapod robot for inspection of airframe components oriented by deep learning technique

  • Technical Paper
  • Published:
Journal of the Brazilian Society of Mechanical Sciences and Engineering Aims and scope Submit manuscript

Abstract

The global competition in the manufacturing industry is becoming more and more aggressive each day. The technologies of Industry 4.0, based on the Internet of Things (IoT), have been pursued in Research and Development, manufacturing, and management processes. In this way, the research consolidated in this paper aims to extend the use of nature-inspired robots in aircraft manufacturing, exploiting the state-of-art technologies and their benefits for productive purposes. This research presents an integrated robotic solution for the inspection of fastened structural joints by a hexapod crawler robot, equipped with a vision sensor, embedded systems, managed by a deep learning algorithm and coordinated in the cloud that moves on the surface of an aircraft providing real-time monitoring via mobile devices. A case study regarding the inspection of airframe fasteners was carried out to demonstrate the application of the proposed solution, the developed method, and its tasks. The automation of the inspection process strives to increase efficiency, cost reduction, streamline ergonomics issues, and support aircraft fabricators. This novel proposal looks for an innovative application in the aeronautical sector based on state-of-art technology faced by intelligent manufacturing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Source: Polek [41] and b Structural assembly of the fuselage. Source: Adapted from Kitchen [29]

Fig. 2
Fig. 3

Source: Guizilini and Ramos [20]

Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. der Aalst V, Bichler WMP, Heinzl M (2018) Robotic process automation. Bus Inf Syst Eng 60:269–272

    Article  Google Scholar 

  2. Aguirre S, Alejandro R (2017) Automation of a business process using robotic process automation (RPA): a case study 8:65–71

  3. Aparna K, Salunke G (2013) Insect inspired hexapd robot for terrain navigation

  4. Arel I, Rose DC, Karnowski TP et al (2010) Deep machine learning—a new frontier in artificial intelligence research. IEEE Comput Intell Mag 5(4):13–18

    Article  Google Scholar 

  5. Barbosa G, Hernandes AC, Luz S, Batista J, Nunes VA, Becker M (2017) A conceptual study towards delivery of consumable materials to aircraft assem- bly stations performed by mobile robots based on industry 4.0 principles. J Aeronaut Aerospace Eng 06

  6. Barbosa GF, Carvalho J, Filho EVG (2014) A proper framework for design of aircraft production system based on lean manufacturing principles focusing to automated processes. Int J Adv Manuf Technol 72(9):1257–1273. https://doi.org/10.1007/s00170-014-5729-3

    Article  Google Scholar 

  7. Bartelsman EJ, Mark D (2000) Understanding Productivity: Lessons from Longitudinal Microdata. J Econ Literat 38(3):569–594. https://doi.org/10.1257/jel.38.3.569

    Article  Google Scholar 

  8. Belter D, Skrzypczynski P (2010) A biologically inspired approach to feasible gait learning for a hexapod robot. Appl Math Comput Sci 20:69–84

    MATH  Google Scholar 

  9. Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828

    Article  Google Scholar 

  10. Bouarfa S, Doğru A, Arizar R, Aydoğan R, Serafico J (2020) Towards automated aircraft maintenance inspection. A use case of detecting aircraft dents using Mask R-CNN. In: AIAA Scitech 2020 forum, p 0389

  11. Caggiano A, Roberto T (2018) Digital factory technologies for robotic au-tomation and enhanced manufacturing cell design. Cogent Eng 5(1):1426676. https://doi.org/10.1080/23311916.2018.1426676

    Article  Google Scholar 

  12. Camacho J, Veiga F, Penalva ML, Diez-Olivan A, Deitert L, López de Lacalle N (2019) On-line monitoring of blind fastener installation process. Materials 12(7):1157

    Article  Google Scholar 

  13. Chu W, Yingguang L, Changqing L, Wenping M, Limin T (2016) Collab-orative manufacturing of aircraft structural parts based on machining features and software agents. Int J Adv Manuf Technol 1421–1434

  14. Cire¸san D, Ueli M, Jonathan M, Luca MG, Ju¨rgen S (2011) High-performance neural networks for visual object classification. Comput Res Reposit CORR

  15. Deng L, Dong Y (2014) Deep learning: methods and applications. Found Trends® Signal Process 7(3–4):197–387. https://doi.org/10.1561/2000000039

  16. Doğru A, Bouarfa S, Arizar R, Aydoğan R (2020) Using convolutional neural networks to automate aircraft maintenance visual inspection. Aerospace 7(12):171

    Article  Google Scholar 

  17. Fang J, Dinesh P, Arjun D, Craig S, Kristofer P (2018) Decentralized control of a hexapod robot using a wireless time synchronized network

  18. Gonzalez-Rodriguez AG, Antonio G-R, Fernando C- G (2014) Improving the energy efficiency and speed of walking robots. Mechatronics 24(5):476–488. http://www.sciencedirect.com/science/article/pii/S0957415814000865

  19. Guizilin VC (2017) Tensorclues: library TensorFlow. https://bitbucket.org/vguizilini/tensorclues.git

  20. Guizilini V, Ramos F (2018) Learning to reconstruct 3D structures for Occu-pancy mapping. Int J Robot Res 37:1595–1609

    Article  Google Scholar 

  21. Guo Y, Yu L, Ard O, Songyang L, Song W, Michael SL (2016) Deep learning for visual understanding. Neurocomputing 187:27–48. https://doi.org/10.1016/j.neucom.2015.09.116

    Article  Google Scholar 

  22. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313:504–507

    Article  MathSciNet  Google Scholar 

  23. Hou B, Yan R (2018) Convolutional auto-encoder based deep feature learning for finger-vein verification. In: 2018 IEEE international symposium on medical measurements and applications (MeMeA), 1–5

  24. Huynh T-C, Jae-Hyung P, Hyung-Jo J, Jeong-Tae K (2019) Quasi-autonomous bolt-loosening detection method using vision-based deep learning and image processing. Autom Constr 105:102844. http://www.sciencedirect.com/science/article/pii/S092658051930250X

  25. Jefferson TG, Richard C, Tony S, Svetan R (2013) Review of reconfigurable assembly systems technologies for cost effective wing structure assem-bly. In: SAE Technical Paper, 09. SAE International. https://doi.org/10.4271/2013-01-2336

  26. Kagermann H, Wolfgang W, Johannes H (2013) Recommendations for implementing the strategic initiative INDUSTRIE 4.0—securing the Fu- ture of German manufacturing industry. Final report of the industrie 4.0 work- ing group. Mu¨nchen: acatech—National Academy of Science and Engineering. http://forschungsunion.de/pdf/industrie40finalreport.pdf

  27. Khan A, Anabia S, Umme Z, Aqsa SQ (2020) A survey of the recent architectures of deep convolutional neural networks. Artif Intel Rev

  28. Khan S, Takehisa Y (2018) A review on the application of deep learning in system health management. Mech Syst Signal Process 107:241–265. http://www.sciencedirect.com/science/article/pii/S0888327017306064

  29. Kitchen ND (2001) Reinforcement system for aircraft fuselage frame and aluminum skin. U.S. Patent 6:286–785 B

  30. Leit˜ao P, Armando WC, Stamatis K (2016) Industrial automation based on cyber-physical systems technologies: prototype implementations and challenges. Comput Ind 81:11–25. Emerging ICT concepts for smart, safe and sustainable industrial systems, http://www.sciencedirect.com/science/article/pii/S0166361515300348

  31. Lewinger WA, Quinn RD (2010) A hexapod walks over irregular terrain using a controller adapted from an insect’s nervous system. In: 2010 IEEE/RSJ international conference on intelligent robots and systems, 3386–3391

  32. Liu C, Pingyu J (2016) A cyber-physical system architecture in shop floor for intelligent manufacturing. Procedia CIRP 56:372–377. In: The 9th international conference on digital enterprise technology—intelligent manufacturing in the knowledge economy era. http://www.sciencedirect.com/science/article/pii/S2212827116310514

  33. Liu Y, Xun X (2016) Industry 4.0 and cloud manufacturing: a comparative analysis. J Manuf Sci Eng 139(3):034701. https://doi.org/10.1115/1.4034667

    Article  Google Scholar 

  34. Masci J, Ueli M, Dan C, Ju¨rgen S (2011) Stacked convolutional auto-encoders for hierarchical feature extraction. In: Timo H, W-lodzis-law D, Mark G, Samuel K (Eds.), Artificial neural networks and machine learning—ICANN 2011, 52–59, Berlin, Heidelberg: Springer

  35. Mitchell TM (1999) Machine learning and data mining. Commun ACM 42(11):30–36

    Article  Google Scholar 

  36. Moayedi H et al (2020) Soft computing method for predicting pressure drop reduction in crude oil pipelines based on machine learning methods. J Braz Soc Mech Sci Eng 42:562

    Article  Google Scholar 

  37. Muelaner J (2019) Unsettled technology domains in robotics for automation in aerospace manufacturing. https://doi.org/10.4271/EPR2019010

  38. Navvabi H, Markazi AH (2019) New AFSMC method for nonlinear system with state-dependent uncertainty: application to hexapod robot position control. J Intel Robot Syst 95(1):61–75. https://doi.org/10.1007/s10846-018-0850-4

    Article  Google Scholar 

  39. Nunes VA, Gustavo FB (2020) Simulation- based analysis of AGV workload used on aircraft manufacturing system: a theoretical approach. Acta Scientiarum Technol 42(1):e47034. http://periodicos.uem.br/ojs/index.php/ActaSciTechnol/article/view/47034

  40. Pahwa RS, Chao J, Paul J, Li Y, Lay New MT, Xie S, James A, Ambikapathi A, Zeng Z, Chandrasekhar VR (2019) FaultNet: faulty rail-valves detection using deep learning and computer vision. In: 2019 IEEE intelligent transportation systems conference (ITSC), Oct, 559–566

  41. Polek G (2014) Airbus starts final assembly of A320neo. https://www.ainonline.com/aviation-news/2014-03-17/airbus-starts-final-assembly-a320neo. Accessed 09 May 2020

  42. Ponti M, Arthur AC, Fabio RJ, Gabriel BPC, Adimara C, Kalinka RLJCB (2016) Precision agriculture: using low-cost systems to acquire low-altitude images. IEEE Comput Graph Appl 36(4):14–20

    Article  Google Scholar 

  43. Ranzato MA, Fu H, Boureau Y-L, Yann L (2007) Unsupervised learning of invariant feature hierarchies with applications to object recognition 7:1–8

  44. Ribeiro M, Andre EL, Heitor SL (2018) A study of deep convolutional auto-encoders for anomaly detection in videos. Pattern Recogn Lett Mach Learn Appl Artif Intel. 105:13–22

  45. Rice M, Li L, Ying G, Wan M, Lim ET, Feng G, Bab VS (2018) Automating the visual inspection of aircraft. In: Aerospace technology and engineering conference

  46. Saranli U, Martin B, Daniel EK (2001) RHex: a simple and highly mobile hexapod robot. Int J Robot Res 20(7):616–631. http://dblp.uni-trier.de/db/journals/ijrr/ijrr20.htmlSaranliBK01

  47. Sinkar A, Aditya P, Chaitanya M, Karthick SSJ, Akash K, Palani IA (2018) Design and development of wall climbing hexapod robot with SMA actuated suction gripper. Procedia Comput Sci. In: International conference on robotics and smart manufacturing (RoSMa2018) 133:222–229. http://www.sciencedirect.com/science/article/pii/S1877050918309712

  48. Stamm M, Krüger P, Pfeiffer H, Köhler B, Reynaert J, Wevers M (2021) In-plane heatwave thermography as digital inspection technique for fasteners in aircraft fuselage panels. Appl Sci 11(1):132

    Article  Google Scholar 

  49. Stehman S (1997) Selecting and interpreting measures of thematic classification accuracy. Remote Sens Environ 62:77–89

    Article  Google Scholar 

  50. Su X, Tomovic M, Zhu D (2019) Diagnosis of gradual faults in high-speed gear pairs using machine learning. J Braz Soc Mech Sci Eng 41:195

    Article  Google Scholar 

  51. Swift T (1991) Repairs to damage tolerant aircraft. In: Atluri SN, Sampath SG, Pin T (Eds). Structural integrity of aging airplanes. Berlin, Heidelberg: Springer, 433–483

  52. Syverson C (2011) What determines productivity? J Econ Lit 49(2):326–65. https://www.aeaweb.org/articles?id=10.1257/jel.49.2.326

  53. Takahashi R, Takashi M, Kuniaki U (2018) Ricap: random image crop- ping and patching data augmentation for deep cnns. In: Asian conference on machine learning, 786–798. PMLR

  54. Tao F, Qi Q, Liu A (2018) Data-driven smart manufacturing. J Manuf Syst 48:157–169

    Article  Google Scholar 

  55. Turchenko V, Eric C, Artur L (2017) A deep convolutional auto- encoder with pooling-unpooling layers in caffe. arXiv preprint arrXiv:1701.04949

  56. Wagels C, Schmitt R (2012) Benchmarking of methods and instruments for self- optimization in future production systems. Procedia CIRP 3:161–166

    Article  Google Scholar 

  57. Wang K, Huang Y, Gong L, Cai C, Zhang Y (2019a) State-wise LSTM-GRU method for ball screw prediction. In: 2019 IEEE aerospacee conference, March, 1–8

  58. Wang P, Ananya RY, Robert XG (2017) Virtualization and deep recognition for system fault classification. J Manuf Syst 44:310–316. Special issue on latest advancements in manufacturing systems at NAMRC 45, http://www.sciencedirect.com/science/article/pii/S0278612517300511

  59. Wang Z, Li H, Zhang X (2019b) Construction waste recycling robot for nails and screws: computer vision technology and neural network approach. Autom Constr 97:220–228

    Article  Google Scholar 

  60. Waurzyniak P (2013) Design and experiment of a robot automatic drilling system. In: Aerospace Automation picks up the pace, 03. Robotic drilling, fastening systems gain ground in aerospace/defense as metrology technologies boost robot accuracy

  61. Wei X, Ziming Y, Yuxin L, Dehua W, Limin J, Yujie L (2019) Rail- way track fastener defect detection based on image processing and deep learning tech- niques: a comparative study. Eng Appl Artif Intel 80:66–81. http://www.sciencedirect.com/science/article/pii/S0952197619300089

  62. Xie Q, Lu D, Huang A, Yang J, Li D, Zhang Y, Wang J (2020) RRCNet: rivet region classification network for rivet flush measurement based on 3-D point cloud. IEEE Trans Instrum Meas 70:1–12

    Google Scholar 

  63. Yao Y (2014) Design and experiment of a robot automatic drilling system. In: SAE Technical Paper, 09. SAE International. https://doi.org/10.4271/2014-01-2246

  64. Zhang Q, Laurence TY, Zhikui C, Peng L (2018) A survey on deep learning for big data. Inf Fusion 42:146–157. http://www.sciencedirect.com/science/article/pii/S1566253517305328

  65. Zhang Y, Xiaowei S, Kenneth L, Wensheng S, Zhigang X, Xuefeng Z (2019) Autonomous bolt loosening detection using deep learning. Struct Health Monit 147592171983750

  66. Zhao X, Yang Z, Niannian W (2019) Bolt loosening angle detection tech- nology using deep learning. Struct Control Health Monit 26(1):e2292. E2292 STC-18–0200.R1. https://doi.org/10.1002/stc.2292.

  67. Zheng P, Honghui W, Zhiqian S, Ray YZ, Yongkui L, Chao L, Khamdi M, Shiqiang Y, Xun X (2018) Smart manufacturing systems for Industry 40: conceptual framework, scenarios, and future perspectives. Front Mech Eng 13(2):137–150. https://doi.org/10.1007/s11465-018-0499-5

    Article  Google Scholar 

  68. Zhong R, George H, Dai QY, Zhou K, Ting Q, Hu GJ (2011) RFID-enabled real-time manufacturing execution system for discrete manufacturing: software design and implementation 5:311–316

  69. Zidek K, Alexander H, Jan P, Slavomir (2019) Recognition of assembly parts by convolutional neural networks. In: Sergej H, Dagmar K, Grzegorz MK, Somnath C, Lucie R (Eds.) Advances in manufacturing engineering and materials Springer: Cham, 281–289

Download references

Acknowledgments

The first, third and fourth authors would like to acknowledge Coordination for the Improvement of Higher Education Personnel (CAPES) for financial support. The second author would like to thank the National Council for Scientific and Technological Development (CNPq) for his technological productivity fellowship (process 314516/2018-2).

Author information

Authors and Affiliations

  1. Federal University of São Carlos—UFSCar, São Carlos, SP, Brazil

    Kelen C. Teixeira Vivaldini, Gustavo Franco Barbosa & Igor Araujo Dias Santos

  2. Federal University of São Carlos—UFSCar, Sorocaba, SP, Brazil

    Pedro H. C. Kim

  3. Kennesaw State, University, Kennesaw, GA, USA

    Grayson McMichael & David A. Guerra-Zubiaga

Authors
  1. Kelen C. Teixeira Vivaldini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gustavo Franco Barbosa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Igor Araujo Dias Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pedro H. C. Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Grayson McMichael
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. David A. Guerra-Zubiaga
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gustavo Franco Barbosa.

Additional information

Technical Editor: Rogerio Sales Gonclaves.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Teixeira Vivaldini, K.C., Franco Barbosa, G., Santos, I.A.D. et al. An intelligent hexapod robot for inspection of airframe components oriented by deep learning technique. J Braz. Soc. Mech. Sci. Eng. 43, 494 (2021). https://doi.org/10.1007/s40430-021-03219-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s40430-021-03219-7

Keywords

Search

Navigation