Abstract
We present a hybrid camera system for capturing video at high spatial and spectral resolutions. Composed of an red, green, and blue (RGB) video camera, a grayscale video camera and a few optical elements, the hybrid camera system simultaneously records two video streams: an RGB video with high spatial resolution, and a multispectral (MS) video with low spatial resolution. After registration of the two video streams, our system propagates the MS information into the RGB video to produce a video with both high spectral and spatial resolution. This propagation between videos is guided by color similarity of pixels in the spectral domain, proximity in the spatial domain, and the consistent color of each scene point in the temporal domain. The propagation algorithm, based on trilateral filtering, is designed to rapidly generate output video from the captured data at frame rates fast enough for real-time video analysis tasks such as tracking and surveillance. We evaluate the proposed system using both simulations with ground truth data and on real-world scenes. The accuracy of spectral capture is examined through comparisons with ground truth and with a commercial spectrometer. The utility of this high resolution MS video data is demonstrated on the applications of dynamic white balance adjustment, object tracking, and separating the appearance contributions of different illumination sources. The various high resolution MS video datasets that we captured will be made publicly available to facilitate research on dynamic spectral data analysis.
Similar content being viewed by others
References
Angelopoulou, E. (2001). Understanding the color of human skin. SPIE Conference on Human Vision and Electronic Imaging, 4299, 243–251.
Brady, D. J., & Gehm, M. E. (2006). Compressive imaging spectrometers using coded apertures. In SPIE (Vol. 6246).
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In ECCV.
Buchsbaum, G. (1980). A spatial processor model for object colour perception. Journal of the Franklin Institute, 310, 1–26.
Cao, X., Du, H., Tong, X., Dai, Q., & Lin, S. (2011a). A prism-mask system for multispectral video acquisition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(12), 2423–2435.
Cao, X., Tong, X., Dai, Q., & Lin, S. (2011b). High resolution multispectral video capture with a hybrid camera system. In 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, pp. 297–304.
Chi, C., Yoo, H., & Ben-Ezra, M. (2010). Multi-spectral imaging by optimized wide band illumination. International Journal of Computer Vision, 86(2), 140–151.
Darling, B. A., Ferwerda, J. A., Berns, R. S., & Chen, T. (2011). Real-time multispectral rendering with complex illumination. In Color and Imaging Conference (Vol. 2011, pp. 345–351). Springfield, VA: Society for Imaging Science and Technology.
Descour, M., & Dereniak, E. (1995). Computed-tomography imaging spectrometer: Experimental calibration and reconstruction results. Applied Optics, 34, 4817–4826.
Du, H., Tong, X., Cao, X., & Lin, S. (2009). A prism-based system for multispectral video acquisition. In Proceedings of the ICCV.
Fletcher-Holmes, D. W., & Harvey, A. R. (2005). Real-time imaging with a hyperspectral fovea. Journal of Optics A: Pure and Applied Optics, 7, S298–S302.
Habel, R., Kudenov, M., & Wimmer, M. (2012). Practical spectral photography. Computer Graphics Forum, 31, 449–458. Wiley Online Library.
Hagen, N., & Dereniak, E. L. (2008). Analysis of computed tomographic imaging spectrometers. I. Spatial and spectral resolution. Applied Optics, 47, F85–F95.
Han, S., Sato, I., Okabe, T., & Sato, Y. (2010). Fast spectral reflectance recovery using DLP projector. In Computer Vision: ACCV, 2010 (pp. 323–335).
James, J. (2007). Spectrograph design fundamentals. Cambridge, MA: Cambridge University Press.
Johnson, W. R., Wilson, D. W., & Bearman, G. (2006). Spatial–spectral modulating snapshot hyperspectral imager. Applied Optics, 45, 1898–1908.
Kim, M. H., Rushmeier, H., Dorsey, J., Harvey, T. A., Prum, R. O., Kittle, D. S., et al. (2012). 3D imaging spectroscopy for measuring hyperspectral patterns on solid objects. ACM Transactions on Graphics (TOG), 31(4), 38.
Kittle, D. S., Marks, D. L., & Brady, D. J. (2012). Design and fabrication of an ultraviolet-visible coded aperture snapshot spectral imager. Optical Engineering, 51(7), 071403-1.
Mooney, J. M., Vickers, V. E., An, M., & Brodzik, A. K. (1997). High throughput hyperspectral infrared camera. Journal of Optical Society of America A, 14, 2951–2961.
Mrozack, A., Marks, D. L., & Brady, D. J. (2012). Coded aperture spectroscopy with denoising through sparsity. Optics Express, 20(3), 2297–2309.
Nvidia, C. (2007). Compute unified device architecture programming guide. Santa Clara, CA: NVIDIA Corp.
Park, J. I., Lee, M. H., Grossberg, M. D., & Nayar, S. K. (2007). Multispectral imaging using multiplexed illumination. In ICCV.
Ross, D., Lim, J., Lin, R. S., & Yang, M. H. (2008). Incremental learning for robust visual tracking. IJCV, 77, 125–141.
Schechner, Y. Y., & Nayar, S. K. (2002). Generalized mosaicing: Wide field of view multispectral imaging. IEEE PAMI, 24(10), 1334–1348.
Shen, L., Tan, P., & Lin, S. (2008). Intrinsic image decomposition with non-local texture cues. In IEEE Conference on Computer Vision and Pattern Recognition, 2008 (CVPR 2008), Anchorage, AK, pp. 1–7.
Smits, B. (1999). An RGB to spectrum conversion for reflectances. Journal of Graphics Tools, 4(4), 11–22.
University of Joensuu Color Group. (2005). Spectral database. http://spectral.joensuu.fi/.
Vandervlugt, C., Masterson, H., Hagen, N., & Dereniak, E. L. (2007). Reconfigurable liquid crystal dispersing element for a computed tomography imaging spectrometer. In SPIE (Vol. 6565).
Volin, C. (2000). MWIR spectrometer operating theory. Tucson, AZ: University of Arizona Press.
Wagadarikar, A., John, R., Willett, R., & Brady, D. (2008). Single disperser design for coded aperture snapshot spectral imaging. Applied Optics, 47, B44–B51.
Wagadarikar, A., Pitsianis, N., Sun, X., & Brady, D. (2009). Video rate spectral imaging using a coded aperture snapshot spectral imager. Optics Express, 17(8), 6368–6388.
Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., & Debevec, P. (2005). Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Transactions on Graphics (TOG), 24(3), 756–764.
Yamaguchi, M., Haneishi, H., Fukuda, H., Kishimoto, J., Kanazawa, H., Tsuchida, M., et al. (2006). High-fidelity video and still-image communication based on spectral information: Natural vision system and its applications. In SPIE/IS &T Electronic Imaging (Vol. 6062).
Yang, Q., Tan, K. H., & Ahuja, N. (2009). Real-time o(1) bilateral filtering. In CVPR.
Acknowledgments
The authors thank Yanxiang Lan, Hao Du and Moshe Ben-Ezra for helpful discussions on implementation issues, Tao Yue for assistance in experimentation, the National Basic Research Project (No. 2010CB731800) and the Key Project of NSFC (No. 61035002), The NSFC Grant No. 61371166, and the reviewers for their constructive comments.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ma, C., Cao, X., Tong, X. et al. Acquisition of High Spatial and Spectral Resolution Video with a Hybrid Camera System. Int J Comput Vis 110, 141–155 (2014). https://doi.org/10.1007/s11263-013-0690-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-013-0690-4