International Journal of Computer Vision

, Volume 110, Issue 2, pp 141–155 | Cite as

Acquisition of High Spatial and Spectral Resolution Video with a Hybrid Camera System

  • Chenguang Ma
  • Xun Cao
  • Xin Tong
  • Qionghai Dai
  • Stephen Lin


We present a hybrid camera system for capturing video at high spatial and spectral resolutions. Composed of an red, green, and blue (RGB) video camera, a grayscale video camera and a few optical elements, the hybrid camera system simultaneously records two video streams: an RGB video with high spatial resolution, and a multispectral (MS) video with low spatial resolution. After registration of the two video streams, our system propagates the MS information into the RGB video to produce a video with both high spectral and spatial resolution. This propagation between videos is guided by color similarity of pixels in the spectral domain, proximity in the spatial domain, and the consistent color of each scene point in the temporal domain. The propagation algorithm, based on trilateral filtering, is designed to rapidly generate output video from the captured data at frame rates fast enough for real-time video analysis tasks such as tracking and surveillance. We evaluate the proposed system using both simulations with ground truth data and on real-world scenes. The accuracy of spectral capture is examined through comparisons with ground truth and with a commercial spectrometer. The utility of this high resolution MS video data is demonstrated on the applications of dynamic white balance adjustment, object tracking, and separating the appearance contributions of different illumination sources. The various high resolution MS video datasets that we captured will be made publicly available to facilitate research on dynamic spectral data analysis.


Multispectral video capture Hybrid camera system 


  1. Angelopoulou, E. (2001). Understanding the color of human skin. SPIE Conference on Human Vision and Electronic Imaging, 4299, 243–251.CrossRefGoogle Scholar
  2. Brady, D. J., & Gehm, M. E. (2006). Compressive imaging spectrometers using coded apertures. In SPIE (Vol. 6246).Google Scholar
  3. Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In ECCV.Google Scholar
  4. Buchsbaum, G. (1980). A spatial processor model for object colour perception. Journal of the Franklin Institute, 310, 1–26.CrossRefGoogle Scholar
  5. Cao, X., Du, H., Tong, X., Dai, Q., & Lin, S. (2011a). A prism-mask system for multispectral video acquisition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(12), 2423–2435.CrossRefGoogle Scholar
  6. Cao, X., Tong, X., Dai, Q., & Lin, S. (2011b). High resolution multispectral video capture with a hybrid camera system. In 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, pp. 297–304.Google Scholar
  7. Chi, C., Yoo, H., & Ben-Ezra, M. (2010). Multi-spectral imaging by optimized wide band illumination. International Journal of Computer Vision, 86(2), 140–151.CrossRefGoogle Scholar
  8. Darling, B. A., Ferwerda, J. A., Berns, R. S., & Chen, T. (2011). Real-time multispectral rendering with complex illumination. In Color and Imaging Conference (Vol. 2011, pp. 345–351). Springfield, VA: Society for Imaging Science and Technology.Google Scholar
  9. Descour, M., & Dereniak, E. (1995). Computed-tomography imaging spectrometer: Experimental calibration and reconstruction results. Applied Optics, 34, 4817–4826.CrossRefGoogle Scholar
  10. Du, H., Tong, X., Cao, X., & Lin, S. (2009). A prism-based system for multispectral video acquisition. In Proceedings of the ICCV.Google Scholar
  11. Fletcher-Holmes, D. W., & Harvey, A. R. (2005). Real-time imaging with a hyperspectral fovea. Journal of Optics A: Pure and Applied Optics, 7, S298–S302.CrossRefGoogle Scholar
  12. Habel, R., Kudenov, M., & Wimmer, M. (2012). Practical spectral photography. Computer Graphics Forum, 31, 449–458. Wiley Online Library.CrossRefGoogle Scholar
  13. Hagen, N., & Dereniak, E. L. (2008). Analysis of computed tomographic imaging spectrometers. I. Spatial and spectral resolution. Applied Optics, 47, F85–F95.CrossRefGoogle Scholar
  14. Han, S., Sato, I., Okabe, T., & Sato, Y. (2010). Fast spectral reflectance recovery using DLP projector. In Computer Vision: ACCV, 2010 (pp. 323–335).Google Scholar
  15. James, J. (2007). Spectrograph design fundamentals. Cambridge, MA: Cambridge University Press.CrossRefGoogle Scholar
  16. Johnson, W. R., Wilson, D. W., & Bearman, G. (2006). Spatial–spectral modulating snapshot hyperspectral imager. Applied Optics, 45, 1898–1908.CrossRefGoogle Scholar
  17. Kim, M. H., Rushmeier, H., Dorsey, J., Harvey, T. A., Prum, R. O., Kittle, D. S., et al. (2012). 3D imaging spectroscopy for measuring hyperspectral patterns on solid objects. ACM Transactions on Graphics (TOG), 31(4), 38.Google Scholar
  18. Kittle, D. S., Marks, D. L., & Brady, D. J. (2012). Design and fabrication of an ultraviolet-visible coded aperture snapshot spectral imager. Optical Engineering, 51(7), 071403-1.Google Scholar
  19. Mooney, J. M., Vickers, V. E., An, M., & Brodzik, A. K. (1997). High throughput hyperspectral infrared camera. Journal of Optical Society of America A, 14, 2951–2961.CrossRefGoogle Scholar
  20. Mrozack, A., Marks, D. L., & Brady, D. J. (2012). Coded aperture spectroscopy with denoising through sparsity. Optics Express, 20(3), 2297–2309.CrossRefGoogle Scholar
  21. Nvidia, C. (2007). Compute unified device architecture programming guide. Santa Clara, CA: NVIDIA Corp.Google Scholar
  22. Park, J. I., Lee, M. H., Grossberg, M. D., & Nayar, S. K. (2007). Multispectral imaging using multiplexed illumination. In ICCV. Google Scholar
  23. Ross, D., Lim, J., Lin, R. S., & Yang, M. H. (2008). Incremental learning for robust visual tracking. IJCV, 77, 125–141.Google Scholar
  24. Schechner, Y. Y., & Nayar, S. K. (2002). Generalized mosaicing: Wide field of view multispectral imaging. IEEE PAMI, 24(10), 1334–1348.CrossRefGoogle Scholar
  25. Shen, L., Tan, P., & Lin, S. (2008). Intrinsic image decomposition with non-local texture cues. In IEEE Conference on Computer Vision and Pattern Recognition, 2008 (CVPR 2008), Anchorage, AK, pp. 1–7.Google Scholar
  26. Smits, B. (1999). An RGB to spectrum conversion for reflectances. Journal of Graphics Tools, 4(4), 11–22.MathSciNetCrossRefGoogle Scholar
  27. University of Joensuu Color Group. (2005). Spectral database.
  28. Vandervlugt, C., Masterson, H., Hagen, N., & Dereniak, E. L. (2007). Reconfigurable liquid crystal dispersing element for a computed tomography imaging spectrometer. In SPIE (Vol. 6565).Google Scholar
  29. Volin, C. (2000). MWIR spectrometer operating theory. Tucson, AZ: University of Arizona Press.Google Scholar
  30. Wagadarikar, A., John, R., Willett, R., & Brady, D. (2008). Single disperser design for coded aperture snapshot spectral imaging. Applied Optics, 47, B44–B51.CrossRefGoogle Scholar
  31. Wagadarikar, A., Pitsianis, N., Sun, X., & Brady, D. (2009). Video rate spectral imaging using a coded aperture snapshot spectral imager. Optics Express, 17(8), 6368–6388.CrossRefGoogle Scholar
  32. Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., & Debevec, P. (2005). Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Transactions on Graphics (TOG), 24(3), 756–764.CrossRefGoogle Scholar
  33. Yamaguchi, M., Haneishi, H., Fukuda, H., Kishimoto, J., Kanazawa, H., Tsuchida, M., et al. (2006). High-fidelity video and still-image communication based on spectral information: Natural vision system and its applications. In SPIE/IS &T Electronic Imaging (Vol. 6062).Google Scholar
  34. Yang, Q., Tan, K. H., & Ahuja, N. (2009). Real-time o(1) bilateral filtering. In CVPR.Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Chenguang Ma
    • 1
  • Xun Cao
    • 2
  • Xin Tong
    • 3
  • Qionghai Dai
    • 1
  • Stephen Lin
    • 3
  1. 1.Department of AutomationTsinghua UniversityBeijingChina
  2. 2.School of Electronic Science and EngineeringNanjing UniversityNanjingChina
  3. 3.Microsoft Research AsiaBeijingChina

Personalised recommendations