Skip to main content
Log in

Acquisition of High Spatial and Spectral Resolution Video with a Hybrid Camera System

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

We present a hybrid camera system for capturing video at high spatial and spectral resolutions. Composed of an red, green, and blue (RGB) video camera, a grayscale video camera and a few optical elements, the hybrid camera system simultaneously records two video streams: an RGB video with high spatial resolution, and a multispectral (MS) video with low spatial resolution. After registration of the two video streams, our system propagates the MS information into the RGB video to produce a video with both high spectral and spatial resolution. This propagation between videos is guided by color similarity of pixels in the spectral domain, proximity in the spatial domain, and the consistent color of each scene point in the temporal domain. The propagation algorithm, based on trilateral filtering, is designed to rapidly generate output video from the captured data at frame rates fast enough for real-time video analysis tasks such as tracking and surveillance. We evaluate the proposed system using both simulations with ground truth data and on real-world scenes. The accuracy of spectral capture is examined through comparisons with ground truth and with a commercial spectrometer. The utility of this high resolution MS video data is demonstrated on the applications of dynamic white balance adjustment, object tracking, and separating the appearance contributions of different illumination sources. The various high resolution MS video datasets that we captured will be made publicly available to facilitate research on dynamic spectral data analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  • Angelopoulou, E. (2001). Understanding the color of human skin. SPIE Conference on Human Vision and Electronic Imaging, 4299, 243–251.

    Article  Google Scholar 

  • Brady, D. J., & Gehm, M. E. (2006). Compressive imaging spectrometers using coded apertures. In SPIE (Vol. 6246).

  • Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In ECCV.

  • Buchsbaum, G. (1980). A spatial processor model for object colour perception. Journal of the Franklin Institute, 310, 1–26.

    Article  Google Scholar 

  • Cao, X., Du, H., Tong, X., Dai, Q., & Lin, S. (2011a). A prism-mask system for multispectral video acquisition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(12), 2423–2435.

    Article  Google Scholar 

  • Cao, X., Tong, X., Dai, Q., & Lin, S. (2011b). High resolution multispectral video capture with a hybrid camera system. In 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, pp. 297–304.

  • Chi, C., Yoo, H., & Ben-Ezra, M. (2010). Multi-spectral imaging by optimized wide band illumination. International Journal of Computer Vision, 86(2), 140–151.

    Article  Google Scholar 

  • Darling, B. A., Ferwerda, J. A., Berns, R. S., & Chen, T. (2011). Real-time multispectral rendering with complex illumination. In Color and Imaging Conference (Vol. 2011, pp. 345–351). Springfield, VA: Society for Imaging Science and Technology.

  • Descour, M., & Dereniak, E. (1995). Computed-tomography imaging spectrometer: Experimental calibration and reconstruction results. Applied Optics, 34, 4817–4826.

    Article  Google Scholar 

  • Du, H., Tong, X., Cao, X., & Lin, S. (2009). A prism-based system for multispectral video acquisition. In Proceedings of the ICCV.

  • Fletcher-Holmes, D. W., & Harvey, A. R. (2005). Real-time imaging with a hyperspectral fovea. Journal of Optics A: Pure and Applied Optics, 7, S298–S302.

    Article  Google Scholar 

  • Habel, R., Kudenov, M., & Wimmer, M. (2012). Practical spectral photography. Computer Graphics Forum, 31, 449–458. Wiley Online Library.

    Article  Google Scholar 

  • Hagen, N., & Dereniak, E. L. (2008). Analysis of computed tomographic imaging spectrometers. I. Spatial and spectral resolution. Applied Optics, 47, F85–F95.

    Article  Google Scholar 

  • Han, S., Sato, I., Okabe, T., & Sato, Y. (2010). Fast spectral reflectance recovery using DLP projector. In Computer Vision: ACCV, 2010 (pp. 323–335).

  • James, J. (2007). Spectrograph design fundamentals. Cambridge, MA: Cambridge University Press.

    Book  Google Scholar 

  • Johnson, W. R., Wilson, D. W., & Bearman, G. (2006). Spatial–spectral modulating snapshot hyperspectral imager. Applied Optics, 45, 1898–1908.

    Article  Google Scholar 

  • Kim, M. H., Rushmeier, H., Dorsey, J., Harvey, T. A., Prum, R. O., Kittle, D. S., et al. (2012). 3D imaging spectroscopy for measuring hyperspectral patterns on solid objects. ACM Transactions on Graphics (TOG), 31(4), 38.

    Google Scholar 

  • Kittle, D. S., Marks, D. L., & Brady, D. J. (2012). Design and fabrication of an ultraviolet-visible coded aperture snapshot spectral imager. Optical Engineering, 51(7), 071403-1.

    Google Scholar 

  • Mooney, J. M., Vickers, V. E., An, M., & Brodzik, A. K. (1997). High throughput hyperspectral infrared camera. Journal of Optical Society of America A, 14, 2951–2961.

    Article  Google Scholar 

  • Mrozack, A., Marks, D. L., & Brady, D. J. (2012). Coded aperture spectroscopy with denoising through sparsity. Optics Express, 20(3), 2297–2309.

    Article  Google Scholar 

  • Nvidia, C. (2007). Compute unified device architecture programming guide. Santa Clara, CA: NVIDIA Corp.

  • Park, J. I., Lee, M. H., Grossberg, M. D., & Nayar, S. K. (2007). Multispectral imaging using multiplexed illumination. In ICCV.

  • Ross, D., Lim, J., Lin, R. S., & Yang, M. H. (2008). Incremental learning for robust visual tracking. IJCV, 77, 125–141.

    Google Scholar 

  • Schechner, Y. Y., & Nayar, S. K. (2002). Generalized mosaicing: Wide field of view multispectral imaging. IEEE PAMI, 24(10), 1334–1348.

    Article  Google Scholar 

  • Shen, L., Tan, P., & Lin, S. (2008). Intrinsic image decomposition with non-local texture cues. In IEEE Conference on Computer Vision and Pattern Recognition, 2008 (CVPR 2008), Anchorage, AK, pp. 1–7.

  • Smits, B. (1999). An RGB to spectrum conversion for reflectances. Journal of Graphics Tools, 4(4), 11–22.

    Article  MathSciNet  Google Scholar 

  • University of Joensuu Color Group. (2005). Spectral database. http://spectral.joensuu.fi/.

  • Vandervlugt, C., Masterson, H., Hagen, N., & Dereniak, E. L. (2007). Reconfigurable liquid crystal dispersing element for a computed tomography imaging spectrometer. In SPIE (Vol. 6565).

  • Volin, C. (2000). MWIR spectrometer operating theory. Tucson, AZ: University of Arizona Press.

    Google Scholar 

  • Wagadarikar, A., John, R., Willett, R., & Brady, D. (2008). Single disperser design for coded aperture snapshot spectral imaging. Applied Optics, 47, B44–B51.

    Article  Google Scholar 

  • Wagadarikar, A., Pitsianis, N., Sun, X., & Brady, D. (2009). Video rate spectral imaging using a coded aperture snapshot spectral imager. Optics Express, 17(8), 6368–6388.

    Article  Google Scholar 

  • Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., & Debevec, P. (2005). Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Transactions on Graphics (TOG), 24(3), 756–764.

    Article  Google Scholar 

  • Yamaguchi, M., Haneishi, H., Fukuda, H., Kishimoto, J., Kanazawa, H., Tsuchida, M., et al. (2006). High-fidelity video and still-image communication based on spectral information: Natural vision system and its applications. In SPIE/IS &T Electronic Imaging (Vol. 6062).

  • Yang, Q., Tan, K. H., & Ahuja, N. (2009). Real-time o(1) bilateral filtering. In CVPR.

Download references

Acknowledgments

The authors thank Yanxiang Lan, Hao Du and Moshe Ben-Ezra for helpful discussions on implementation issues, Tao Yue for assistance in experimentation, the National Basic Research Project (No. 2010CB731800) and the Key Project of NSFC (No. 61035002), The NSFC Grant No. 61371166, and the reviewers for their constructive comments.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xun Cao.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ma, C., Cao, X., Tong, X. et al. Acquisition of High Spatial and Spectral Resolution Video with a Hybrid Camera System. Int J Comput Vis 110, 141–155 (2014). https://doi.org/10.1007/s11263-013-0690-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-013-0690-4

Keywords

Navigation