A visible-light and infrared video database for performance evaluation of video/image fusion methods
- 212 Downloads
In general, the fusion of visible-light and infrared images produces a composite representation where both data are pictured in a single image. The successful development of image/video fusion algorithms relies on realistic infrared/visible-light datasets. To the best of our knowledge, there is a particular shortage of databases with registered and synchronized videos from the infrared and visible-light spectra suitable for image/video fusion research. To address this need we recorded an image/video fusion database using infrared and visible-light cameras under varying illumination conditions. Moreover, different scenarios have been defined to better challenge the fusion methods, with various contexts and contents providing a wide variety of meaningful data for fusion purposes, including non-planar scenes, where objects appear on different depth planes. However, there are several difficulties in creating datasets for research in infrared/visible-light image fusion. Camera calibration, registration, and synchronization can be listed as important steps of this task. In particular, image registration between imagery from sensors of different spectral bands imposes additional difficulties, as it is very challenging to solve the correspondence problem between such images. Motivated by these challenges, this work introduces a novel spatiotemporal video registration method capable of generating registered and temporally aligned infrared/visible-light video sequences. The proposed workflow improves the registration accuracy when compared to the state-of-the art. By applying the proposed methodology to the recorded database we have generated the visible-light and infrared video database for image fusion, a publicly available database to be used by the research community to test and benchmark fusion schemes.
KeywordsInfrared/visible image/video database Image registration Image fusion Camera calibration
This work was partially supported by CAPES/Pro-Defesa under Grant No. 23038.009094/2013-83.
- Bouguet, J.-Y. (2012). Camera calibration toolbox for Matlab. http://www.vision.caltech.edu/bouguetj/calib_doc/. Accessed December 12, 2012.
- Bradski, G., Kaehler, A., & Pisarevsky, V. (2005). Learning-based computer vision with intel’s open source computer vision library. Intel Technology Journal, 9, 119–130.Google Scholar
- Caspi, Y., & Irani, M. (2000). A step towards sequence-to-sequence alignment. In Proceedings of the 2000 IEEE conference on computer vision and pattern recognition (Vol. 2, pp. 682–689).Google Scholar
- Datta, A., Kim, J.-S., & Kanade, T. (2009). Accurate camera calibration using iterative refinement of control points. In Proceedings of the 2009 IEEE international conference on computer vision workshops (ICCV workshops) (pp. 1201–1208).Google Scholar
- Debevec, P. E., Taylor, C. J., & Malik, J. (1996). Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proceedings of the 23rd annual conference on computer graphics and interactive techniques (pp. 11–20).Google Scholar
- Ellmauthaler, A., da Silva, E. A. B., Pagliari, C. L., & Gois, J. N. (2013a). A robust temporal alignment technique for infrared and visible-light video sequences. In XXXI Simp. Brasileiro de Telecomunicações.Google Scholar
- Ellmauthaler, A., da Silva, E., Pagliari, C., Gois, J., & Neves, S. R. (2013b). A novel iterative calibration approach for thermal infrared cameras. In Proceedings of the 2013 IEEE international conference on image processing.Google Scholar
- Ellmauthaler, A., da Silva, E., Pagliari, C., & Neves, S. (2012). Infrared-visible image fusion using the undecimated wavelet transform with spectral factorization and target extraction. In Proceedings of the 2012 IEEE international conference on image processing (pp. 2661–2664).Google Scholar
- Faugeras, O. (1993). Three dimensional computer vision: A geometric viewpoint. Cambridge: The MIT Press.Google Scholar
- Gschwandtner, M., Kwitt, R., Uhl, A., & Pree, W. (2011). Infrared camera calibration for dense depth map construction. In Proceedings of the 2011 intelligent vehicles symposium (pp. 857–862).Google Scholar
- Heikkilä, J., & Silvén, O. (1997). A four-step camera calibration procedure with implicit image correction. In Proceedings of the 1997 IEEE computer society conference on computer vision and pattern recognition (pp. 1106–1112).Google Scholar
- http://adas.cvc.uab.es/projects/simeve/. Accessed January 11, 2015.
- http://vcipl-okstate.org/pbvs/bench/. Accessed January 11, 2015.
- http://www.cis.rit.edu/pelz/scanpaths/data/bristol-eden.htm. Accessed January 11, 2015.
- http://www.ino.ca/en/video-analytics-dataset/. Accessed February 08, 2017.
- http://www.polymtl.ca/litiv/vid/. Accessed January 11, 2015.
- http://www.vap.aau.dk/stereo-thermal-dataset/. Accessed February 08, 2017.
- Int. society on information fusion. http://isif.org. Accessed April 07, 2015.
- Lewis, J. J., et al. (2006). The Eden Project multi-sensor data set. In Technical report TR-UoB-WS-Eden-Project-Data-Set (University of Bristol and Waterfall Solutions Ltd, London).Google Scholar
- Loop, C., & Zhang, Z. (1999). Computing rectifying homographies for stereo vision. In IEEE computer society conference on computer vision and pattern recognition (Vol. 1).Google Scholar
- Morris, N. J. W., Avidan, S., Matusik, W., & Pfister, H. (2007). Statistics of infrared images. In Proceedings of the 2007 IEEE computer society conference on computer vision and pattern recognition (pp. 1–7).Google Scholar
- Morris, N. J. W., Avidan, S., & Pfister, H. Statistics of infrared images. http://www.dgp.toronto.edu/~nmorris/IR/. Accessed January 11, 2015.
- Prakash, C., & Karam, L. (2012). Camera calibration using adaptive segmentation and ellipse fitting for localizing control points. In Proceedings of the 2012 IEEE international conference on image processing (pp. 341–344).Google Scholar
- Prakash, S., Lee, P., Caelli, T., & Raupach, T. (2006). Robust thermal camera calibration and 3d mapping of object surface temperatures. In Proceedings of the XXVIII SPIE conference on thermosense (Vol. 6205).Google Scholar
- Saponaro, P., Sorensen, S., Rhein, S., & Kambhamettu, C. (2015). Improving calibration of thermal stereo cameras using heated calibration board. In Proceedings of the 2015 IEEE international conference on image processing.Google Scholar
- St-Charles, P. -L., Bilodeau, G.-A., & Bergevin, R. (2015). Online multimodal video registration based on shape matching. In 2015 IEEE conference on computer vision and pattern recognition workshops (CVPRW) (pp. 26–34).Google Scholar
- Ukrainitz, Y., & Irani, M. (2006). Aligning sequences and actions by maximizing space-time correlations. In Proceedings of the 2006 European conference on computer vision (Vol. 3953, pp. 538–550).Google Scholar
- Visible-light and infrared video database for image fusion. http://www.smt.ufrj.br/~fusion/. Accessed November 11, 2017.
- Zhang, M. M. E. A. (2015). Vais: A dataset for recognizing maritime imagery in the visible and infrared spectrums. In The IEEE conference on computer vision and pattern recognition (CVPR) workshops.Google Scholar
- Zhang, Z., & Hanson, A. (1996). 3D reconstruction based on homography mapping. In ARPA image understanding workshop (pp. 249–399).Google Scholar