Skip to main content
SpringerLink
Log in

An integrated system for automated 3D visualization and monitoring of vehicles

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Recent technological advances in computer vision and the advent of commercial RGB-D sensors have certainly boosted 3D modeling applications. This work presents an integrated system that enables the digitization of big objects, like vehicles, with low-cost RGB-D sensors. The implemented system can be used for visualization and monitoring of vehicles in large fleets that currently require a time-consuming manual inspection process. The main objective is to achieve an efficient consolidation of multiple views of a vehicle inside a moving frame, to acquire color and depth data and generate its 3D representation. The proposed integrated system denoises the acquired depth maps, aligns the produced point clouds captured in different time instances, and builds the 3D-reconstructed mesh. Finally, we apply a texture mapping algorithm to acquire realistic texture details and remove any visible seams. We evaluate all modules of the implemented system by performing several experiments with scanned vehicles.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Discher S, Richter R, Döllner J (2019) Concepts and techniques for web-based visualization and processing of massive 3d point clouds with semantics. Graph Model 104:101036

    Article  Google Scholar 

  2. Christen M, Nebiker S (2015) Visualisation of complex 3d city models on mobile webbrowsers using cloud-based image provisioning. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences

  3. Aivaliotis P, Zampetis A, Michalos G, Makris S (2017) A machine learning approach for visual recognition of complex parts in robotic manipulation. Procedia Manufacturing 11:423–430

    Article  Google Scholar 

  4. Wang M, Harrisson S, Destarac M, Marty J-D (2019) Measuring cloud point pressures by image analysis: a simple and reproducible alternative method to direct visual determination. The Journal of Supercritical Fluids 152:104572

    Article  Google Scholar 

  5. Stavropoulos G, Moschonas P, Moustakas K, Tzovaras D, Strintzis MG (2010) 3-d model search and retrieval from range images using salient features. IEEE Trans Multimed 12(7):692–704

    Article  Google Scholar 

  6. Gai J, Tang L, Steward BL (2019) Automated crop plant detection based on the fusion of color and depth images for robotic weed control. Journal of Field Robotics 37(1):35–52

    Article  Google Scholar 

  7. Rougier C, Auvinet E, Rousseau J, Mignotte M, Meunier J (2011) Fall detection from depth map video sequences. In: Toward useful services for elderly and people with disabilities. Springer, Berlin, pp 121–128

  8. Yang L, Ren Y, Zhang W (2016) 3d depth image analysis for indoor fall detection of elderly people. Digital Commun Netw 2(1):24–34

    Article  Google Scholar 

  9. Giakoumis D, Drosou A, Cipresso P, Tzovaras D, Hassapis G, Gaggioli A, Riva G (2012) Using activity-related behavioural features towards more effective automatic stress detection. PLoS ONE 7 (9):e43571

    Article  Google Scholar 

  10. Dimitriou N, Leontaris L, Vafeiadis T, Ioannidis D, Wotherspoon T, Tinker G, Tzovaras D (2020) Fault diagnosis in microelectronics attachment via deep learning analysis of 3d laser scans. IEEE Trans Ind Electron 67(7):5748–5757

    Article  Google Scholar 

  11. Dimitriou N, Leontaris L, Vafeiadis T, Ioannidis D, Wotherspoon T, Tinker G, Tzovaras D (2020) A deep learning framework for simulation and defect prediction applied in microelectronics. Simul Model Pract Theory 100:102063

    Article  Google Scholar 

  12. Rousopoulou V, Papachristou K, Dimitriou N, Drosou A, Tzovaras D (2019) Automated mechanical multi-sensorial scanning. In: International conference on computer vision systems (ICVS)

  13. Semeniuta O, Dransfeld S, Falkman P (2016) Vision-based robotic system for picking and inspection of small automotive components. In: 2016 IEEE international conference on automation science and engineering (CASE). IEEE

  14. Besl PJ, McKay ND (1992) A method for registration of 3-d shapes. IEEE Trans Pattern Anal Mach Intell 14(2):239–256

    Article  Google Scholar 

  15. Pomerleau F, Colas F, Siegwart R, Magnenat S (2013) Comparing ICP variants on real-world data sets. Auton Robot 34(3):133–148

    Article  Google Scholar 

  16. Rusinkiewicz S, Levoy M (2001) Efficient variants of the icp algorithm. In: 3Dim, vol 1, pp 145–152

  17. Bouaziz S, Tagliasacchi A, Pauly M (2013) Sparse iterative closest point. In: Proceedings of the eleventh eurographics/ACMSIGGRAPH symposium on geometry processing. Eurographics Association, pp 113–123

  18. Gelfand N, Mitra NJ, Guibas LJ, Pottmann H (2005) Robust global registration. In: Symposium on geometry processing, vol 2. Vienna, Austria, p 5

  19. Glover J, Bradski G, Rusu RB (2012) Monte Carlo pose estimation with quaternion kernels and the Bingham distribution. In: Robotics: science and systems, vol 7, pp 97

  20. Makadia A, Patterson A, Daniilidis K (2006) Fully automatic registration of 3d point clouds. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06), vol 1, pp 1297–1304. IEEE

  21. Ovsjanikov M, Mérigot Q, Mémoli F, Guibas L (2010) One point isometric matching with the heat kernel. In: Computer graphics forum, vol 29, pp 1555–1564. Wiley Online Library

  22. Rusu RB, Blodow N, Beetz M (2009) Fast point feature histograms (FPFH) for 3d registration. In: 2009 IEEE international conference on robotics and automation, pp 3212–3217. IEEE

  23. Zhou Q-Y, Park J, Koltun V (2016) Fast global registration. In: European conference on computer vision, pp 766–782. Springer

  24. Zeng A, Song S, Nießner M, Fisher M, Xiao J, Funkhouser T (2017) 3dmatch: learning local geometric descriptors from RGB-D reconstructions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pP 1802–1811

  25. Deng H, Birdal T, Ilic S (2018) Ppfnet: global context aware local features for robust 3d point matching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 195–205

  26. Aoki Y, Goforth H, Srivatsan RA, Lucey S (2019) Pointnetlk: robust & efficient point cloud registration using Pointnet. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7163–7172

  27. Qi CR, Su H, Mo K, Guibas LJ (2017) Pointnet: deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 652–660

  28. Lucas BD, Kanade T et al (1981) An iterative image registration technique with an application to stereo vision

  29. Cazals F, Giesen J (2006) Delaunay triangulation based surface reconstruction. In: Effective computational geometry for curves and surfaces, pp 231–276. Springer

  30. Carr JC, Beatson RK, Cherrie JB, Mitchell TJ, Fright WR, McCallum BC, Evans TR (2001) Reconstruction and representation of 3d objects with radial basis functions. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp 67–76. ACM

  31. Kazhdan M, Bolitho M, Hoppe H (2006) Poisson surface reconstruction. In: Proceedings of the fourth eurographics symposium on geometry processing, vol 7

  32. Kazhdan M, Hoppe H (2013) Screened Poisson surface reconstruction. ACM Trans Graph (ToG) 32(3):29

    Article  Google Scholar 

  33. Slavcheva M, Baust M, Ilic S (2018) Sobolevfusion: 3d reconstruction of scenes undergoing free non-rigid motion. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2646–2655

  34. Allène C, Pons J-P, Keriven R (2008) Seamless image-based texture atlases using multi-band blending. In: 2008 19th international conference on pattern recognition, pp 1–4. IEEE

  35. Kehl W, Navab N, Ilic S (2014) Coloured signed distance fields for full 3d object reconstruction. In: BMVC

  36. Lempitsky V, Ivanov D (2007) Seamless mosaicing of image-based texture maps. In: 2007 IEEE conference on computer vision and pattern recognition, pp 1–6. IEEE

  37. Waechter M, Moehrle N, Goesele M (2014) Let there be color! large-scale texturing of 3d reconstructions. In: European conference on computer vision, pp 836–850. Springer

  38. Pérez P, Gangnet M, Blake A (2003) Poisson image editing. ACM Trans Graphics (TOG) 22(3):313–318

    Article  Google Scholar 

  39. Papachristou K, Dimitriou N, Drosou A, Karagiannis G, Tzovaras D (2018) Realistic texture reconstruction incorporating spectrophotometric color correction. In: 2018 25th IEEE international conference on image processing (ICIP), pp 415–419. IEEE

  40. Hartley R, Zisserman A (2003) Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  41. Bradski G (2000) The OpenCV library. Dr. Dobb’s Journal of Software Tools

  42. Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330– 1334

    Article  Google Scholar 

  43. Yan S, Wu C, Wang L, Xu F, An L, Guo K, Liu Y (2018) Ddrnet: depth map denoising and refinement for consumer depth cameras using cascaded CNNs. In: Proceedings of the European conference on computer vision (ECCV), pp 151–167

  44. Haefner B, Quéau Y, Möllenhoff T, Cremers D (2018) Fight ill-posedness with ill-posedness: single-shot variational depth super-resolution from shading. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 164– 174

  45. Heo M, Lee J, Kim K-R, Kim H-U, Kim C-S (2018) Monocular depth estimation using whole strip masking and reliability-based refinement. In: The European conference on computer vision. ECCV

  46. Zhang Y, Khamis S, Rhemann C, Valentin J, Kowdle A, Tankovich V, Schoenberg M, Izadi S, Funkhouser T, Fanello S (2018) Activestereonet: end-to-end self-supervised learning for active stereo systems. In: The European conference on computer vision (ECCV)

  47. Kleitsiotis I, Dimitriou N, Konstantinos V, Tzovaras D (2019) Color-guided adaptive support weights for active stereo systems. In: International conference on computer vision systems. ICVS

  48. Khamis S, Fanello S, Rhemann C, Kowdle A, Valentin J, Izadi S (2018) Stereonet: guided hierarchical refinement for real-time edge-aware depth prediction. In: The european conference on computer vision. ECCV

  49. Tzovaras D, Karagiannis N, Strintzis MG (1998) Robust image watermarking in the subband or discrete cosine transform domain. In: 9th european signal processing conference (EUSIPCO 1998), pp 1–4. IEEE

  50. Padilla-López JR, Chaaraoui AA, Flórez-Revuelta F (2015) Visual privacy protection methods: a survey. Expert Syst Appl 42(9):4177–4195

    Article  Google Scholar 

  51. Hu P, Ramanan D (2017) Finding tiny faces. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 951–959

  52. Yang S, Luo P, Loy C-C, Tang X (2016) Wider face: a face detection benchmark. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5525– 5533

  53. Revaud J, Weinzaepfel P, Harchaoui Z, Schmid C (2016) Deepmatching: hierarchical deformable dense matching. Int J Comput Vis 120(3):300–323

    Article  MathSciNet  Google Scholar 

  54. Guerrero P, Kleiman Y, Ovsjanikov M, Mitra NJ (2018) Pcpnet learning local shape properties from raw point clouds. In: Computer graphics forum. Wiley Online Library, vol 37, pp 75–85

  55. Salamanis A, Kehagias DD, Filelis-Papadopoulos CK, Tzovaras D, Gravvanis GA (2015) Managing spatial graph dependencies in large volumes of traffic data for travel-time prediction. IEEE Trans Intell Transp Syst 17(6):1678–1687

    Article  Google Scholar 

Download references

Funding

This work has been co-funded by the European Union and the General Secretariat of Research and Technology, Ministry of Development & Investments, under the project INVIVO/T2DGE-0951 of the Bilateral S&T Cooperation Program Greece - Germany 2017. This work has also been partially supported by the European Commission through project RECLAIM funded by the European Union H2020 programme under Grant Agreement no. 869884.

Author information

Authors and Affiliations

  1. Centre for Research & Technology Hellas, Information Technologies Institute, Thessaloniki, Thermi, 57001, Greece

    Stella Bounareli, Ioannis Kleitsiotis, Lampros Leontaris, Nikolaos Dimitriou, Konstantinos Votis & Dimitrios Tzovaras

  2. iKnowHow, Kifisias 116 Ave & Davaki 1 Str, Athens, GR, 11526, Greece

    Aggeliki Pilalitou, Nikolaos Valmantonis & Efthymios Pachos

Authors
  1. Stella Bounareli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ioannis Kleitsiotis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lampros Leontaris
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nikolaos Dimitriou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aggeliki Pilalitou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Nikolaos Valmantonis
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Efthymios Pachos
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Konstantinos Votis
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Dimitrios Tzovaras
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lampros Leontaris.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bounareli, S., Kleitsiotis, I., Leontaris, L. et al. An integrated system for automated 3D visualization and monitoring of vehicles. Int J Adv Manuf Technol 111, 1797–1809 (2020). https://doi.org/10.1007/s00170-020-06148-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-020-06148-2

Keywords

Search

Navigation