Skip to main content
SpringerLink
Log in

Co-Registration of 2D Imagery and 3D Point Cloud Data

  • Chapter
  • First Online:
Reconstruction and Analysis of 3D Scenes

  • 2549 Accesses

Abstract

In this chapter, we address the fact that particularly thermal information offers many advantages for scene analysis, since people may easily be detected as heat sources in typical indoor or outdoor environments and, furthermore, a variety of concealed objects such as heating pipes as well as structural properties such as defects in isolation may be observed. Additionally, a 3D mapping involving a range camera with high frame rate and a thermal camera with typical video frame rate may be helpful to describe the evolution of a dynamic 3D scene over time. In order to achieve a respective 3D mapping, we present a novel and fully automatic framework consisting of four successive components: (i) a radiometric correction, (ii) a geometric calibration, (iii) a robust approach for detecting reliable feature correspondences, and (iv) a co-registration of 3D point cloud data and thermal information. For the last component, we consider two different approaches represented by a RANSAC-based homography estimation for almost planar scenes and a RANSAC-based projective scan matching technique for general scenes. For the example of an indoor scene, we demonstrate the performance of our framework in terms of both accuracy and applicability. We additionally show that efficient straightforward techniques allow a sharpening of the blurry thermal infrared information or a categorization of the acquired data with respect to background, people, passive scene manipulation, and active scene manipulation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Alba MI, Barazzetti L, Scaioni M, Rosina E, Previtali M (2011) Mapping infrared data on terrestrial laser scanning 3D models of buildings. Remote Sens 3(9):1847–1870

    Article  Google Scholar 

  2. Bai X, Zhou F, Xue B (2011) Fusion of infrared and visual images through region extraction by using multi scale center-surround top-hat transform. Opt Express 19(9):8444–8457

    Article  Google Scholar 

  3. Balaras CA, Argiriou AA (2002) Infrared thermography for building diagnostics. Energy Build 34(2):171–183

    Article  Google Scholar 

  4. Bay H, Tuytelaars T, Van Gool L (2006) SURF: speeded up robust features. In: Proceedings of the European conference on computer vision, vol 1, pp 404–417

    Google Scholar 

  5. Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-up robust features (SURF). Comput Vis Image Underst 110(3):346–359

    Article  Google Scholar 

  6. Borrmann D, Afzal H, Elseberg J, Nüchter A (2012) Mutual calibration for 3D thermal mapping. In: Proceedings of the international IFAC symposium on robot control, pp 605–610

    Google Scholar 

  7. Borrmann D, Elseberg J, Nüchter A (2012) Thermal 3D mapping of building façades. In: Proceedings of the international conference on intelligent autonomous systems, pp 173–182

    Google Scholar 

  8. Bouguet J-Y (2010) Camera calibration toolbox for Matlab. Computer Vision Research Group, Department of Electrical Engineering, California Institute of Technology, Pasadena, USA. http://www.vision.caltech.edu/bouguetj/calib_doc/index.html. Accessed 30 May 2015

  9. Brown DC (1971) Close-range camera calibration. Photogramm Eng 37(8):855–866

    Google Scholar 

  10. Chen S, Leung H (2009) An EM-CI based approach to fusion of IR and visual images. In: Proceedings of the international conference on information fusion, pp 1325–1330

    Google Scholar 

  11. Coiras E, Santamaría J, Miravet C (2000) Segment-based registration technique for visual-IR images. Opt Eng 39(1):282–289

    Article  Google Scholar 

  12. Droeschel D, Holz D, Behnke S (2010) Multi-frequency phase unwrapping for time-of-flight cameras. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, pp 1463–1469

    Google Scholar 

  13. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  14. Hartley RI, Zisserman A (2008) Multiple view geometry in computer vision. University Press, Cambridge

    MATH  Google Scholar 

  15. Heikkilä J, Silvén O (1997) A four-step camera calibration procedure with implicit image correction. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 1106–1112

    Google Scholar 

  16. Hoegner L (2014) Automatische Texturierung von Fassaden aus terrestrischen Infrarot-Bildsequenzen. PhD thesis, Institut für Photogrammetrie und Kartographie, Technische Universität München, München, Germany

    Google Scholar 

  17. Hoegner L, Kumke H, Meng L, Stilla U (2007) Automatic extraction of textures from infrared image sequences and database integration for 3D building models. PFG—Photogramm Fernerkund Geoinf 6(2007):459–468

    Google Scholar 

  18. Hoegner L, Kumke H, Schwarz A, Meng L, Stilla U (2007) Strategies for texturing building models with low resolution infrared image sequences. Int Arch Photogramm Remote Sens Spat Inf Sci XXXVI-5/C55:1–6

    Google Scholar 

  19. Hoegner L, Roth L, Weinmann M, Jutzi B, Hinz S, Stilla U (2014) Fusion von Time-of-Flight-Entfernungsdaten und thermalen IR-Bildern. AVN—Allg Vermess-Nachr 5(2014):192–197

    Google Scholar 

  20. Iwaszczuk D, Hoegner L, Stilla U (2011) Detection of windows in IR building textures using masked correlation. In: Stilla U, Rottensteiner F, Mayer H, Jutzi B, Butenuth M (eds) Photogrammetric image analysis, ISPRS Conference—Proceedings. Lecture notes in computer science, vol 6952, Springer, Heidelberg, pp 133–146

    Google Scholar 

  21. Jutzi B (2009) Investigations on ambiguity unwrapping of range images. Int Arch Photogramm Remote Sens Spat Inf Sci XXXVIII-3/W8:265–270

    Google Scholar 

  22. Jutzi B (2012) Extending the range measurement capabilities of modulated range imaging devices by time-frequency-multiplexing. AVN—Allg Vermess-Nachr 2(2012):54–62

    Google Scholar 

  23. Lagüela S, González-Jorge H, Armesto J, Arias P (2011) Calibration and verification of thermographic cameras for geometric measurements. Infrared Phys Technol 54(2):92–99

    Article  Google Scholar 

  24. Lagüela S, Martínez J, Armesto J, Arias P (2011) Energy efficiency studies through 3D laser scanning and thermographic technologies. Energy Build 43(6):1216–1221

    Article  Google Scholar 

  25. Lepetit V, Moreno-Noguer F, Fua P (2009) EPnP: an accurate O(n) solution to the PnP problem. Int J Comput Vis 81(2):155–166

    Article  Google Scholar 

  26. Liu L, Stamos I (2005) Automatic 3D to 2D registration for the photorealistic rendering of urban scenes. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, vol 2, pp 137–143

    Google Scholar 

  27. Liu L, Stamos I (2012) A systematic approach for 2D-image to 3D-range registration in urban environments. Comput Vis Image Underst 116(1):25–37

    Article  Google Scholar 

  28. Liu Z, Laganière R (2007) Context enhancement through infrared vision: a modified fusion scheme. Signal Image Video Process 1(4):293–301

    Article  MATH  Google Scholar 

  29. Lowe DG (1991) Fitting parameterized three-dimensional models to images. IEEE Trans Pattern Anal Mach Intell 13(5):441–450

    Article  MathSciNet  Google Scholar 

  30. Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of the IEEE international conference on computer vision, pp 1150–1157

    Google Scholar 

  31. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  32. Luhmann T, Ohm J, Piechel J, Roelfs T (2010) Geometric calibration of thermographic cameras. Int Arch Photogramm Remote Sens Spat Inf Sci XXXVIII-5:411–416

    Google Scholar 

  33. Ma Y, Soatto S, Košecká J, Sastry SS (2005) An invitation to 3-D vision: from images to geometric models. Springer, New York

    MATH  Google Scholar 

  34. Markov S, Birk A (2007) Detecting humans in 2D thermal images by generating 3D models. In: Hertzberg J, Beetz M, Englert R (eds) KI 2007: Advances in Artificial Intelligence. Lecture notes in artificial intelligence, vol 4667, Springer, Heidelberg, pp 293–307

    Google Scholar 

  35. Moreno-Noguer F, Lepetit V, Fua P (2007) Accurate non-iterative O(n) solution to the PnP problem. In: Proceedings of the IEEE international conference on computer vision, pp 1–8

    Google Scholar 

  36. Park C, Bae K-H, Choi S, Jung J-H (2008) Image fusion in infrared image and visual image using normalized mutual information. Proc SPIE 6968:69681Q-1–9

    Google Scholar 

  37. Parmehr EG, Fraser CS, Zhang C, Leach J (2013) Automatic registration of optical imagery with 3D lidar data using local combined mutual information. ISPRS Ann Photogramm Remote Sens Spat Inf Sci II-5/W2:229–234

    Google Scholar 

  38. Pelagotti A, Del Mastio A, Uccheddu F, Remondino F (2009) Automated multispectral texture mapping of 3D models. In: Proceedings of the European signal processing conference, pp 1215–1219

    Google Scholar 

  39. Pulli K, Abi-Rached H, Duchamp T, Shapiro LG, Stuetzle W (1998) Acquisition and visualization of colored 3D objects. In: Proceedings of the international conference on pattern recognition, pp 11–15

    Google Scholar 

  40. Prakash S, Pei YL, Caelli T (2006) 3D mapping of surface temperature using thermal stereo. In: Proceedings of the international conference on control, automation, robotics and vision, pp 1–4

    Google Scholar 

  41. Steger C (2001) Similarity measures for occlusion, clutter, and illumination invariant object recognition. In: Radig B, Florczyk S (eds) Pattern Recognition, DAGM2001. Lecture notes in computer science, vol 2191, Springer, Heidelberg, pp 148–154

    Google Scholar 

  42. Szeliski R (2011) Computer vision: algorithms and applications. Springer, London

    Book  MATH  Google Scholar 

  43. Toth C, Ju H, Grejner-Brzezinska D (2011) Matching between different image domains. In: Stilla U, Rottensteiner F, Mayer H, Jutzi B, Butenuth M (eds) Photogrammetric image analysis, ISPRS Conference—Proceedings. Lecture notes in computer science, vol 6952, Springer, Heidelberg, pp 37–47

    Google Scholar 

  44. Ulrich M (2003) Hierarchical real-time recognition of compound objects in images. PhD thesis, Institut für Photogrammetrie und Kartographie, Technische Universität München, München, Germany

    Google Scholar 

  45. Vidas S, Moghadam P, Bosse M (2013) 3D thermal mapping of building interiors using an RGB-D and thermal camera. In: Proceedings of the IEEE international conference on robotics and automation, pp 2311–2318

    Google Scholar 

  46. Weinmann M (2013) Visual features—From early concepts to modern computer vision. In: Farinella GM, Battiato S, Cipolla R (eds) Advanced topics in computer vision. Advances in computer vision and pattern recognition, Springer, London, pp 1–34

    Google Scholar 

  47. Weinmann M, Jutzi B (2011) Fully automatic image-based registration of unorganized TLS data. Int Arch Photogramm Remote Sens Spat Inf Sci XXXVIII-5/W12:55–60

    Google Scholar 

  48. Weinmann M, Jutzi B (2012) A step towards dynamic scene analysis with active multi-view range imaging systems. Int Arch Photogramm Remote Sens Spat Inf Sci XXXIX-B3:433–438

    Google Scholar 

  49. Weinmann M, Hoegner L, Leitloff J, Stilla U, Hinz S, Jutzi B (2012) Fusing passive and active sensed images to gain infrared-textured 3D models. Int Arch Photogramm Remote Sens Spat Inf Sci XXXIX-B1:71–76

    Google Scholar 

  50. Weinmann M, Leitloff J, Hoegner L, Jutzi B, Stilla U, Hinz S (2014) Thermal 3D mapping for object detection in dynamic scenes. ISPRS Ann Photogramm Remote Sens Spat Inf Sci II-1:53–60

    Google Scholar 

  51. Wu Y, Hu Z (2006) PnP problem revisited. J Math Imaging Vis 24(1):131–141

    Article  MathSciNet  Google Scholar 

  52. Xue Z, Blum RS, Li Y (2002) Fusion of visual and IR images for concealed weapon detection. In: Proceedings of the international conference on image fusion, pp 1198–1205

    Google Scholar 

  53. Yao F, Sekmen A (2008) Multi-source airborne IR and optical image fusion and its application to target detection. In: Proceedings of the international symposium on advances in visual computing, pp 651–660

    Google Scholar 

  54. Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334

    Article  Google Scholar 

  55. Zitová B, Flusser J (2003) Image registration methods: a survey. Image Vis Comput 21(11):977–1000

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Photogrammetry and Remote Sensing, Karlsruhe Institute of Technology, Karlsruhe, Germany

    Martin Weinmann

Authors
  1. Martin Weinmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martin Weinmann .

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Weinmann, M. (2016). Co-Registration of 2D Imagery and 3D Point Cloud Data. In: Reconstruction and Analysis of 3D Scenes. Springer, Cham. https://doi.org/10.1007/978-3-319-29246-5_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-29246-5_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-29244-1

  • Online ISBN: 978-3-319-29246-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation