Advertisement

Journal of Real-Time Image Processing

, Volume 12, Issue 4, pp 697–708 | Cite as

Multi-camera platform for panoramic real-time HDR video construction and rendering

  • Vladan PopovicEmail author
  • Kerem Seyid
  • Eliéva Pignat
  • Ömer Çogal
  • Yusuf Leblebici
Special Issue Paper

Abstract

High dynamic range (HDR) images are usually obtained by capturing several images of the scene at different exposures. Previous HDR video techniques adopted the same principle by stacking HDR frames in time domain. We designed a new multi-camera platform which is able to construct and render HDR panoramic video in real time, with \(1{,}024 \times 256\) resolution and a frame rate of 25 fps. We exploit the overlapping fields of view between the cameras with different exposures to create an HDR radiance map. We propose a method for HDR frame reconstruction which merges the previous HDR imaging techniques with the algorithms for panorama reconstruction. The developed FPGA-based processing system is able to reconstruct the HDR frame using the proposed method and tone map the resulting image using a hardware-adapted global operator. The measured throughput of the system is 245 MB/s, which is, up to our knowledge, among the fastest HDR video processing system.

Keywords

High dynamic range Smart cameras FPGA implementation Tone mapping Real-time systems 

Notes

Acknowledgments

The authors would like to thank H. Afshari, S. Hauser and, P. Bruehlmeier for their work on designing the hardware platform.

Supplementary material

11554_2014_444_MOESM1_ESM.mpeg (17.1 mb)
Supplementary material 1 (MPEG 22882 kb)
11554_2014_444_MOESM2_ESM.mpeg (22.3 mb)
Supplementary material 2 (MPEG 17514 kb)

References

  1. 1.
    Chander, G., Markham, B.L., Helder, D.L.: Summary of current radiometric calibration coefficients for Landsat MSS, TM, ETM+, and EO-1 ALI sensors. Remote Sens. Environ. 113(5), 893–903 (2009)CrossRefGoogle Scholar
  2. 2.
    Jungmann, J.H., MacAleese, L., Visser, J., Vrakking, M.J.J., Heeren, R.M.A.: High dynamic range bio-molecular ion microscopy with the timepix detector. Anal. Chem. 83(20), 7888–7894 (2011)CrossRefGoogle Scholar
  3. 3.
    Bloch, C.: The HDRI handbook 2.0: high dynamic range imaging for photographers and CG artists. Rocky Nook, USA (2013)Google Scholar
  4. 4.
    Mann, S., Picard, R.W.: On being ’Undigital’ with digital cameras: extending dynamic range by combining differently exposed pictures. In: Proceedings of IS&T, pp. 442–448 (1995)Google Scholar
  5. 5.
    Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: ACM SIGGRAPH 97. New York, pp. 369–378 (1997)Google Scholar
  6. 6.
    Ward, G.: Graphics gems II. Real pixels, pp. 80–83. Academic Press, San Diego (1991)CrossRefGoogle Scholar
  7. 7.
    Mitsunaga, T., Nayar, S.: Radiometric self calibration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 374–380 (1999)Google Scholar
  8. 8.
    Pattanaik, S.N., Reinhard, E., Ward, G., Debevec, P.E.: High dynamic range imaging—acquisition, display, and image-based lighting. Morgan Kaufmann, Massachusetts (2005)Google Scholar
  9. 9.
    Robertson, M.A., Borman, S., Stevenson, R.L.: Estimation-theoretic approach to dynamic range enhancement using multiple exposures. J. Electron. Imaging 12(2), 219–228 (2003)CrossRefGoogle Scholar
  10. 10.
    Granados, M., Ajdin, B., Wand, M., Theobalt, C., Seidel, H.P., Lensch, H.P.A.: Optimal HDR reconstruction with linear digital cameras. In: Proceedings of 23rd IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 215–222 (2010)Google Scholar
  11. 11.
    Hasinoff, S.W., Durand, F., Freeman, W.T.: Noise-optimal capture for high dynamic range photography. In: Proceedings of 23rd IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 553–560 (2010)Google Scholar
  12. 12.
    Mertens, T., Kautz, J., Van Reeth, F.: Exposure fusion. In: Pacific Conferece on Computer Graphics and Applications, pp. 382–390. doi: 10.1109/PG.2007.17 (2007)
  13. 13.
    Saleem, A., Beghdadi, A., Boashash, B.: Image fusion-based contrast enhancement. EURASIP J. Image Video Process. 2012, 10 (2012). doi: 10.1186/1687-5281-2012-10 CrossRefGoogle Scholar
  14. 14.
    Martinez-Sanchez, A., Fernandez, C., Navarro, P.J., Iborra, A.: A novel method to increase LinLog CMOS sensors’ performance in high dynamic range scenarios. Sensors 11(9), 8412–8429 (2011). doi: 10.3390/s110908412 CrossRefGoogle Scholar
  15. 15.
    Ward, G., Rushmeier, H., Piatko, C.: A visibility matching tone reproduction operator for high dynamic range scenes. IEEE Trans. Vis. Comput. Graph. 3(4), 291–306 (1997). doi: 10.1109/2945.646233 CrossRefGoogle Scholar
  16. 16.
    Pattanaik, S.N., Tumblin, J., Yee, H., Greenberg, D.P.: Time-dependent visual adaptation for fast realistic image display. In: Proceedings of ACM SIGGRAPH 00. New York, pp. 47–54. doi: 10.1145/344779.344810 (2000)
  17. 17.
    Drago, F., Myszkowski, K., Annen, T., Chiba, N.: Adaptive logarithmic mapping for displaying high contrast scenes. Comput. Graph. forum 22(3), 419–426 (2003). doi: 10.1111/1467-8659.00689 CrossRefGoogle Scholar
  18. 18.
    Mantiuk, R., Daly, S., Kerofsky, L.: Display adaptive tone mapping. ACM Trans. Graph. 27(3), 68:1–68:10 (2008)CrossRefGoogle Scholar
  19. 19.
    Reinhard, E., Stark, M., Shirley, P., Ferwerda, J.: Photographic tone reproduction for digital images. ACM Trans. Graph. 21(3), 267–276 (2002). doi: 10.1145/566654.566575 CrossRefGoogle Scholar
  20. 20.
    Fattal, R., Lischinski, D., Werman, M.: Gradient domain high dynamic range compression. ACM Trans. Graph. 21(3), 249–256 (2002). doi: 10.1145/566654.566573 CrossRefGoogle Scholar
  21. 21.
    Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. 21(3), 257–266 (2002). doi: 10.1145/566654.566574 CrossRefGoogle Scholar
  22. 22.
    Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High dynamic range video. ACM Trans. Graph. 22(3), 319–325 (2003). doi: 10.1145/882262.882270 CrossRefGoogle Scholar
  23. 23.
    Kalantari, N.K., Shechtman, E., Barnes, C., Darabi, S., Goldman, D.B., Sen, P.: Patch-based high dynamic range video. In: ACM Transactions on Graphics (TOG) (Proceedings of SIGGRAPH Asia 2013), vol. 32(6) (2013)Google Scholar
  24. 24.
    Gupta, M., Iso, D., Nayar, S.: Fibonacci exposure bracketing for high dynamic range imaging. In: IEEE International Conference on Computer Vision (ICCV) (2013)Google Scholar
  25. 25.
    Tocci, M.D., Kiser, C., Tocci, N., Sen, P.: A versatile HDR video production system. ACM Trans. Graph. 30(4), 41:1–41:10 (2011). doi: 10.1145/2010324.1964936 CrossRefGoogle Scholar
  26. 26.
    Kronander, J., Gustavson, S., Bonnet, G., Unger, J.: Unified HDR reconstruction from raw CFA data. In: Proceedings of IEEE International Conference on Computational Photography (2013)Google Scholar
  27. 27.
    Ramachandra, V., Zwicker, M., Nguyen, T.: HDR imaging from differently exposed multiview videos. In: IEEE 3DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video, pp. 85–88 (2008)Google Scholar
  28. 28.
    Portz, T., Zhang, L., Jiang, H.: Random coded sampling for high-speed HDR video. In: IEEE International Conference on Computational Photography (ICCP). doi: 10.1109/ICCPhot.6528308 (2013)
  29. 29.
    Hassan, F., Carletta, J.E.: An FPGA-based architecture for a local tone-mapping operator. J. Real Time Image Process. 2(4), 293–308 (2007). doi: 10.1007/s11554-007-0056-7 CrossRefGoogle Scholar
  30. 30.
    Vytla, L., Hassan, F., Carletta, J.: A real-time implementation of gradient domain high dynamic range compression using a local Poisson solver. J. Real Time Image Process. 8(2), 153–167 (2013). doi: 10.1007/s11554-011-0198-5 CrossRefGoogle Scholar
  31. 31.
    Lapray, P.J., Heyrman, B., Rosse, M., Ginhac, D.: HDR-ARtiSt: high dynamic range advanced real-time imaging system. In: IEEE International Symposium on Circuits and Systems, pp. 1428–1431. doi: 10.1109/ISCAS.2012.6271513 (2012)
  32. 32.
    Lapray, P.J., Heyrman, B., Ginhac, D.: HDR-ARtiSt: an adaptive real-time smart camera for high dynamic range imaging. J. Real Time Image Process. 1–16. doi: 10.1007/s11554-013-0393-7 (2014)
  33. 33.
    Akyuz, A.O.: High dynamic range imaging pipeline on the GPU. J. Real Time Image Process. 1–15. doi: 10.1007/s11554-012-0270-9 (2012)
  34. 34.
    Akil, M., Grandpierre, T., Perroton, L.: Real-time dynamic tone-mapping operator on GPU. J. Real Time Image Process. 7(3), 165–172 (2012). doi: 10.1007/s11554-011-0196-7 CrossRefGoogle Scholar
  35. 35.
    Slomp, M., Oliveira, M.M.: Real-time photographic local tone reproduction using summed-area tables, pp. 82–91. Computer Graphics International, Istanbul (2008)Google Scholar
  36. 36.
    Afshari, H.: A real-time multi-aperture omnidirectional visual sensor with interconnected network of smart cameras. Ph.D. thesis, Ecole Polytechnique Federale de Lausanne (EPFL), Switzerland. doi: 10.5075/epfl-thesis-5717 (2013)
  37. 37.
    Brown, M., Lowe, D.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1), 59–73 (2007)CrossRefGoogle Scholar
  38. 38.
    Szeliski, R., Uyttendaele, M., Steedly, D.: Fast poisson blending using multi-splines. In: IEEE International Conference on Computational Photography (ICCP). doi: 10.1109/ICCPHOT.2011.5753119 (2011)
  39. 39.
    Wilburn, B., Joshi, N., Vaish, V., Talvala, E.V., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M.: High performance imaging using large camera arrays. ACM trans. Graph. 24, 765–776 (2005). doi: 10.1145/1073204.1073259 CrossRefGoogle Scholar
  40. 40.
    Popovic, V., Afshari, H., Schmid, A., Leblebici, Y.: Real-time implementation of gaussian image blending in a spherical light field camera. In: Proceedings of IEEE International Conference on Industrial Technology, pp. 1173–1178. doi: 10.1109/ICIT.2013.6505839 (2013)
  41. 41.
    Popovic, V., Seyid, K., Akin, A., Cogal, O., Afshari, H., Schmid, A., Leblebici, Y.: Image blending in a high frame rate FPGA-based multi-camera system. J. Signal Process. Syst. 1–16. doi: 10.1007/s11265-013-0858-8 (2013)
  42. 42.
    Hartley, R.I., Zisserman, A.: Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, Cambridge (2004)CrossRefzbMATHGoogle Scholar
  43. 43.
    de Berg, M., van Kreveld, M., Overmars, M., Schwarzkopf, O.: Computational geometry: algorithms and applications, 2nd edn. Springer, Berlin (2000)CrossRefzbMATHGoogle Scholar
  44. 44.
    Yoshida, A., Blanz, V., Myszkowski, K., Seidel, H.P.: Perceptual evaluation of tone mapping operators with real-world scenes. In: SPIE Human Vision & Electronic Imaging, vol. X, pp. 192–203. doi: 10.1117/12.587782 (2005)
  45. 45.
    Meyer-Baese, U.: Digital signal processing with field programmable gate arrays, 3rd edn. Springer, Berlin (2007)zbMATHGoogle Scholar
  46. 46.
    Popovic, V., Seyid, K., Schmid, A., Leblebici, Y.: Real-time hardware implementation of multi-resolution image blending. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2741–2745. doi: 10.1109/ICASSP.2013.6638155 (2013)

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Vladan Popovic
    • 1
    Email author
  • Kerem Seyid
    • 1
  • Eliéva Pignat
    • 1
  • Ömer Çogal
    • 1
  • Yusuf Leblebici
    • 1
  1. 1.Microelectronic Systems LaboratoryEcole Polytechnique Fédérale de Lausanne (EPFL)LausanneSwitzerland

Personalised recommendations