3D Point Cloud Analysis for Detection and Characterization of Defects on Airplane Exterior Surface

  • Igor Jovančević
  • Huy-Hieu Pham
  • Jean-José Orteu
  • Rémi Gilblas
  • Jacques Harvent
  • Xavier Maurice
  • Ludovic Brèthes


Three-dimensional surface defect inspection remains a challenging task. This paper describes a novel automatic vision-based inspection system that is capable of detecting and characterizing defects on an airplane exterior surface. By analyzing 3D data collected with a 3D scanner, our method aims to identify and extract the information about the undesired defects such as dents, protrusions or scratches based on local surface properties. Surface dents and protrusions are identified as the deviations from an ideal, smooth surface. Given an unorganized point cloud, we first smooth noisy data by using Moving Least Squares algorithm. The curvature and normal information are then estimated at every point in the input data. As a next step, Region Growing segmentation algorithm divides the point cloud into defective and non-defective regions using the local normal and curvature information. Further, the convex hull around each defective region is calculated in order to englobe the suspicious irregularity. Finally, we use our new technique to measure the dimension, depth, and orientation of the defects. We tested and validated our novel approach on real aircraft data obtained from an Airbus A320, for different types of defect. The accuracy of the system is evaluated by comparing the measurements of our approach with ground truth measurements obtained by a high-accuracy measuring device. The result shows that our work is robust, effective and promising for industrial applications.


Aircraft Defect detection Defect characterization Non destructive evaluation 3D scanner Unorganized point cloud 

List of symbols

\(P_{N} = \{p_{1}, p_{2},...,p_{N}\}\)

A set of N points, \(p_{i}\) is the \(i^{th}\) data point

\(p_{i} = (x_{i}, y_{i}, z_{i})\)

A point in three-dimensional

\(P^K= {p_{1}, p_{2},..., p_{K}}\)

The set of points which are located in the k-neighborhood of a query point \(p_{i}\)


The centroid of the data e.g., given a set of points \(P_{N}\), we have: \(\overline{p} = \frac{1}{N}(\sum \limits _{i = 1}^{N}x_{i}, \sum \limits _{i = 1}^{N} y_{i}, \sum \limits _{i = 1}^{N} z_{i})\)


A surface normal estimated at a point \(p_{i}\)

\(\cdot \)

The dot product

\(\times \)

The cross product

\(\Vert \circ \Vert \)

The Euclidean norm of \( \circ \)


  1. 1.
    Benhabiles, H., Lavoué, G., Vandeborre, J., Daoudi, M.: Learning boundary edges for 3d mesh segmentation. Comput. Gr. Forum 30, 2170–2182 (2011)CrossRefGoogle Scholar
  2. 2.
    Besl, P.J., Jain, R.C.: Segmentation through variable-order surface fitting. IEEE Trans. Pattern Anal. Mach. Intell. 10(2), 167–192 (1988)CrossRefGoogle Scholar
  3. 3.
    Bhanu, B., Lee, S., Ho, C., Henderson, T.: Range Data Processing: Representation of Surfaces by Edges. Department of Computer Science, University of Utah, Utah (1985)Google Scholar
  4. 4.
    Borsu, V., Yogeswaran, A., Payeur, P.: Automated surface deformations detection and marking on automotive body panels. In: 2010 IEEE Conference on Automation Science and Engineering (CASE), pp. 551–556. IEEE (2010)Google Scholar
  5. 5.
    Itseez: Open source computer vision library (2015). https://github.com/itseez/opencv
  6. 6.
    Carlbom, I., Paciorek, J.: Planar geometric projections and viewing transformations. ACM Comput. Surv. (CSUR) 10(4), 465–502 (1978)CrossRefMATHGoogle Scholar
  7. 7.
    Deng, H., Zhang, W., Mortensen, E., Dietterich, T., Shapiro, L.: Principal curvature-based region detector for object recognition. In: CVPR’07. IEEE Conference on Computer Vision and Pattern Recognition, p. 18. IEEE (2007)Google Scholar
  8. 8.
    Dey, T.K., Li, G., Sun, J.: Normal estimation for point clouds: a comparison study for a voronoi based method. In: Pointbased graphics, 2005. Eurographics/IEEE VGTC Symposium Proceedings, pp. 39–46. IEEE (2005)Google Scholar
  9. 9.
    Dimitrov, D., Knauer, C., Kriegel, K., Rote, G.: On the bounding boxes obtained by principal component analysis. In: 22nd European Workshop on Computational Geometry, pp. 193–196. Citeseer (2006)Google Scholar
  10. 10.
    Dyn, N., Hormann, K., Kim, S., Levin, D.: Optimizing 3d triangulations using discrete curvature analysis. Math. Methods Curves Surf. 28(5), 135–146 (2001)MATHMathSciNetGoogle Scholar
  11. 11.
    Filin, S.: Surface clustering from airborne laser scanning data. Int. Arch. Photogramme. Remote Sens. Spat. Inf. Sci. 34(3/A), 119–124 (2002)Google Scholar
  12. 12.
    Fua, P., Sander, P.: Segmenting unstructured 3d points into surfaces. In: Computer Vision—ECCV’92, pp. 676–680. Springer (1992)Google Scholar
  13. 13.
    Golovinskiy, A., Funkhouser, T.: Randomized cuts for 3d mesh analysis. ACM Trans. Gr. (TOG) 27(5), 145 (2008)Google Scholar
  14. 14.
    Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 3rd edn, pp. 669–671. Prentice Hall, Upper Saddle River (2002)Google Scholar
  15. 15.
    Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 3rd edn, pp. 667–669. Prentice Hall, Upper Saddle River (2002)Google Scholar
  16. 16.
    Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 3rd edn, pp. 655–657. Prentice Hall, Upper Saddle rRiver (2002)Google Scholar
  17. 17.
    Gottschalk, S., Lin, M.C., Manocha, D.: Obbtree: a hierarchical structure for rapid interference detection. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 171–180. ACM (1996)Google Scholar
  18. 18.
    Haddad, N.A.: From ground surveying to 3d laser scanner: a review of techniques used for spatial documentation of historic sites. J. King Saud Univ. Eng. Sci. 23(2), 109–118 (2011)Google Scholar
  19. 19.
    Hoover, A., JeanBaptiste, G., Jiang, X., Flynn, P.J., Bunke, H., Goldgof, D.B., Bowyer, K., Eggert, D.W., Fitzgibbon, A., Fisher, R.B.: An experimental comparison of range image segmentation algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 18(7), 673–689 (1996)CrossRefGoogle Scholar
  20. 20.
    Jin, H., Yezzi, A.J., Soatto, S.: Region-based segmentation on evolving surfaces with application to 3d reconstruction of shape and piecewise constant radiance. In: Computer Vision—ECCV 2004, pp. 114–125. Springer (2004)Google Scholar
  21. 21.
    Jolliffe, I.: Principal Component Analysis. Wiley Online Library, Hoboken (2002)MATHGoogle Scholar
  22. 22.
    Katz, S., Tal, A.: Hierarchical Mesh Decomposition Using Fuzzy Clustering and Cuts, vol. 22. ACM, New York (2003)Google Scholar
  23. 23.
    Khan, W.: Image segmentation techniques: a survey. J. Image Gr. 1(4), 166–170 (2013)Google Scholar
  24. 24.
    Klasing, K., Althoff, D., Wollherr, D., Buss, M.: Comparison of surface normal estimation methods for range sensing applications. In: ICRA’09. IEEE International Conference on Robotics and Automation, pp. 3206–3211. IEEE (2009)Google Scholar
  25. 25.
    Koenderink, J.J., van Doorn, A.J.: Surface shape and curvature scales. Image Vis. Comput. 10(8), 557–564 (1992)CrossRefGoogle Scholar
  26. 26.
    Köster, K., Spann, M.: Mir: an approach to robust clustering-application to range image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22(5), 430–444 (2000)CrossRefGoogle Scholar
  27. 27.
    Lancaster, P., Salkauskas, K.: Surfaces generated by moving least squares methods. Math. Comput. 37(155), 141–158 (1981)CrossRefMATHMathSciNetGoogle Scholar
  28. 28.
    Levin, D.: The approximation power of moving leasts-quares. Math. Comput. Am. Math. Soc. 67(224), 1517–1531 (1998)CrossRefMATHGoogle Scholar
  29. 29.
    Levin, D.: Mesh-independent surface interpolation. In: Farin, G. (ed.) Geometric Modeling for Scientific Visualization, pp. 37–49. Springer, Berlin (2004)CrossRefGoogle Scholar
  30. 30.
    Marani, R., Roselli, G., Nitti, M., Cicirelli, G., D’Orazio, T., Stella, E.: A 3d vision system for high resolution surface reconstruction. In: 2013 Seventh International Conference on Sensing Technology (ICST), pp. 157–162. IEEE (2013)Google Scholar
  31. 31.
    Mitra, N.J., Nguyen, A.: Estimating surface normals in noisy point cloud data. In: Proceedings of the Nineteenth Annual Symposium on Computational Geometry, pp. 322–328. ACM (2003)Google Scholar
  32. 32.
    Mumtaz, R., Mumtaz, M., Mansoor, A.B., Masood, H.: Computer aided visual inspection of aircraft surfaces. Int. J. Image Process. (IJIP) 6(1), 38 (2012)Google Scholar
  33. 33.
    Nealen, A.: An as-short-as-possible introduction to the least squares, weighted least squares and moving least squares methods for scattered data approximation and interpolation. http://www.nealen.com/projects, pp. 130–150 (2004)
  34. 34.
    Nguyen, A., Le, B.: 3d point cloud segmentation: a survey. In: 2013 6th IEEE Conference on Robotics, Automation and Mechatronics (RAM), pp. 225–230. IEEE (2013)Google Scholar
  35. 35.
    Nurunnabi, A., Belton, D., West, G.: Robust segmentation in laser scanning 3d point cloud data. In: 2012 International Conference on Digital Image Computing Techniques and Applications (DICTA), p. 18. IEEE (2012)Google Scholar
  36. 36.
    Nurunnabi, A., West, G., Belton, D.: Robust methods for feature extraction from mobile laser scanning 3D point clouds. In: Veenendaal, B., Kealy, A. (eds.) Research@Locate’15, pp. 109–120. Brisbane, Australia (2015)Google Scholar
  37. 37.
    Pauling, F., Bosse, M., Zlot, R.: Automatic segmentation of 3d laser point clouds by ellipsoidal region growing. In: Australasian Conference on Robotics and Automation (ACRA) (2009)Google Scholar
  38. 38.
    Pauly, M., Gross, M., Kobbelt, L.P.: Efficient simplification of point-sampled surfaces. In: Proceedings of the Conference on Visualization’02, pp. 163–170. IEEE Computer Society (2002)Google Scholar
  39. 39.
    Peng, J., Li, Q., Kuo, C.J., Zhou, M.: Estimating gaussian curvatures from 3d meshes. In: International Society for Optics and Photonics on Electronic Imaging, pp. 270–280 (2003)Google Scholar
  40. 40.
    Rabbani, T., Van Den Heuvel, F., Vosselmann, G.: Segmentation of point clouds using smoothness constraint. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 36(5), 248–253 (2006)Google Scholar
  41. 41.
    Rusu, R.B.: Semantic 3d object maps for everyday manipulation in human living environments. Ph.D. thesis, Technische Universität München (2009)Google Scholar
  42. 42.
    Rusu, R.B., Cousins, S.: 3D is here: Point Cloud Library (PCL). In: IEEE International Conference on Robotics and Automation (ICRA). Shanghai (2011)Google Scholar
  43. 43.
    Rusu, R.B., Marton, Z.C., Blodow, N., Dolha, M., Beetz, M.: Towards 3d point cloud based object maps for household environments. Robot. Auton. Syst. 56(11), 927–941 (2008)CrossRefGoogle Scholar
  44. 44.
    Sappa, A.D., Devy, M.: Fast range image segmentation by an edge detection strategy. In: Proceedings of Third International Conference on 3D Digital Imaging and Modeling, pp. 292–299. IEEE (2001)Google Scholar
  45. 45.
    Schnabel, R., Wahl, R., Klein, R.: Efficient ransac for point cloud shape detection. Comput. Gr. Forum 26, 214–226 (2007)CrossRefGoogle Scholar
  46. 46.
    Seher, C., Siegel, M., Kaufman, W.M.: Automation tools for nondestructive inspection of aircraft: Promise of technology transfer from the civilian to the military sector. In: Fourth Annual IEEE DualUse Technologies and Applications Conference (1994)Google Scholar
  47. 47.
    Shakarji, C.M., et al.: Leasts-quares fitting algorithms of the nist algorithm testing system. J. Res. Natl. Inst. Stand. Technol. 103, 633–641 (1998)CrossRefGoogle Scholar
  48. 48.
    Siegel, M., Gunatilake, P.: Remote inspection technologies for aircraft skin inspection. In: Proceedings of the 1997 IEEE Workshop on Emergent Technologies and Virtual Systems for Instrumentation and Measurement, Niagara Falls, Canada, pp. 79–78 (1997)Google Scholar
  49. 49.
    Siegel, M., Gunatilake, P., Podnar, G.: Robotic assistants for aircraft inspectors. Ind. Robot 25(6), 389–400 (1998)CrossRefGoogle Scholar
  50. 50.
    Simari, P., Nowrouzezahrai, D., Kalogerakis, E., Singh, K.: Multi-objective shape segmentation and labeling. Comput. Gr. Forum 28, 1415–1425 (2009)CrossRefGoogle Scholar
  51. 51.
    Strom, J., Richardson, A., Olson, E.: Graph-based segmentation for colored 3d laser point clouds. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2131–2136. IEEE (2010)Google Scholar
  52. 52.
    Tang, P., Akinci, B., Huber, D.: Characterization of three algorithms for detecting surface flatness defects from dense point clouds. In: IS&T/SPIE Electronic Imaging on International Society for Optics and Photonics, pp. 72,390N–72,390N (2009)Google Scholar
  53. 53.
    Tóvári, D., Pfeifer, N.: Segmentation based robust interpolation—a new approach to laser data filtering. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 36(3/19), 79–84 (2005)Google Scholar
  54. 54.
    Wang, C., Wang, X., Zhou, X., Li, Z.: The aircraft skin crack inspection based on different-source sensors and support vector machines. J. Nondestruct. Eval. 35(3), 46 (2016). doi:10.1007/s10921-016-0359-3 CrossRefGoogle Scholar
  55. 55.
    Wendland, H.: Piecewise polynomial, positive definite and compactly supported radial functions of minimal degree. Adv. Comput. Math. 4(1), 389–396 (1995)CrossRefMATHMathSciNetGoogle Scholar
  56. 56.
    Wirjadi, O.: Survey of 3D Image Segmentation Methods, vol. 35. ITWM, kaiserslautern (2007)Google Scholar
  57. 57.
    Wong, B.S., Wang, X., Koh, C.M., Tui, C.G., Tan, C., Xu, J.: Crack detection using image processing techniques for radiographic inspection of aircraft wing spar. Insight NonDestructive Test. Cond. Monit. 53(10), 552–556 (2011)Google Scholar
  58. 58.
    Yang, J., Gan, Z., Li, K., Hou, C.: Graph-based segmentation for rgb-d data using 3d geometry enhanced superpixels. IEEE Trans. Cybernet. 45(5), 927–940 (2015)Google Scholar
  59. 59.
    Zhang, X., Li, H., Cheng, Z., Zhang, Y.: Robust curvature estimation and geometry analysis of 3d point cloud surfaces. J. Inf. Comput. Sci 6(5), 1983–1990 (2009)Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.Institut Clément Ader (ICA)Université de Toulouse, CNRS, INSA, UPS, Mines Albi, ISAEAlbiFrance
  2. 2.KEONYSBlagnacFrance

Personalised recommendations