Skip to main content
SpringerLink
Log in

Color HDR video processing architecture for smart camera

How to capture the HDR video in real-time

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

This paper presents a novel FPGA architecture of high dynamic range (HDR) video processing pipeline, based on the capturing of a sequence of differently exposed images. An acquisition process enabling multi-exposure HDR as well as fast implementation of local tone mapping operator involving bilateral filtering is proposed. The HDR acquisition process is enhanced by the application of novel deghosting method, which is dedicated for hardware implementation and proposed in this paper. The hardware processing pipeline is designed with regards to efficiency and performance and the calculations are performed in fixed point arithmetic. The pipeline is suitable for programmable hardware (FPGA—Field Programmable Gate Arrays) implementation and it achieves real-time performance on full HD HDR video which overcomes state-of-the-art solutions that use local tone mapping and deghosting algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

(Image retrieved from dataset [11])

Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. www.arm.com/products/system-ip/amba-specifications.

  2. Short example videos captured by our camera prototype are available at http://www.fit.vutbr.cz/research/groups/graph/downloads/jrtip2018_hdr_video.zip.

References

  1. Baba, N., Isobe, S., Norimoto, Y., Noguchi, M.: Stellar speckle image reconstruction by the shift-and-add method. Appl. Opt. 24, 1403–1405 (1985)

    Article  Google Scholar 

  2. Bouderbane, M., Dubois, J., Heyrman, B., Lapray, P.J., Ginhac, D.: Ghost removing for HDR real-time video stream generation. In: Real-time image and video processing, 98970F (2016)

  3. Chiu, C.T., Wang, T.H., Ke, W.M., Chuang, C.Y., Huang, J.S., Wong, W.S., Tsay, R.S., Wu, C.J.: Real-time tone-mapping processor with integrated photographic and gradient compression using 0.13 um technology on an arm soc platform. J. Signal Process. Syst. 64, 93–107 (2011)

    Article  Google Scholar 

  4. Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: ACM transactions on graphics (TOG), SIGGRAPH ’97, pp. 369–378. ACM, New York (1997)

    Google Scholar 

  5. Drago, F., Myszkowski, K., Annen, T., Chiba, N.: Adaptive logarithmic mapping for displaying high contrast scenes. In: Computer graphics forum, pp. 419-426. Blackwell Publishing, Inc, Oxford, UK (2003)

    Google Scholar 

  6. Duan, J., Bressan, M., Dance, C., Qiu, G.: Tone-mapping high dynamic range images by novel histogram adjustment. Pattern Recogn. 43, 1847–1862 (2010)

    Article  Google Scholar 

  7. Duan, J., Qiu, G., Chen, M.: Comprehensive fast tone mapping for high dynamic range image visualization. In: Pacific Graphics (2005)

  8. Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. In: ACM transactions on graphics (TOG), SIGGRAPH ’02. pp. 257–266. ACM, New York (2002)

    Google Scholar 

  9. Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone and detail manipulation. In: ACM SIGGRAPH 2008 Papers, SIGGRAPH ’08. ACM, New York, NY, USA (2008)

  10. Fattal, R., Lischinski, D., Werman, M.: Gradient domain high dynamic range compression. In: ACM Transactions on Graphics (TOG), SIGGRAPH ’02, pp. 249–256. ACM, New York (2002)

    Google Scholar 

  11. Froehlich, J., Grandinetti, S., Eberhardt, B., Walter, S., Schilling, A., Brendel, H.: Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays. In Proc. SPIE 9023 (2014)

  12. Grosch, T.: Fast and robust high dynamic range image generation with camera and object movement. Vision, Modeling, and Visualization 2006: Proceedings, pp. 277–283 (2006)

  13. Hassan, F., Carletta, J.E.: A real-time FPGA-based architecture for a Reinhard-like tone mapping operator. In: SIGGRAPH/EUROGRAPHICS symposium on Graphics Hardware, pp. 65–71 (2007)

  14. Kiser, C., Reinhard, E., Tocci, M., Tocci, N.: Real time automated tone mapping system for hdr video. In: IEEE Proceedings of the IEEE International Conference on Image Processing (2012)

  15. Lapray, P.J., Heyrman, B., Ginhac, D.: HDR-ARtiSt: A 1280 x 1024-pixel adaptive real-time smart camera for high dynamic range video. In: SPIE Photonics Europe. Brussels, Belgium (2014)

  16. Lapray, P.J., Heyrman, B., Ginhac, D.: Hdr-artist: an adaptive real-time smart camera for high dynamic range imaging. J. Real Time Image Process. 12, 747–762 (2016)

    Article  Google Scholar 

  17. Lapray, P.J., Heyrman, B., Ross, M., Ginhac, D.: HDR-artist: High dynamic range advanced real-time imaging system. In: 2012 IEEE International Symposium on Circuits and Systems (2012)

  18. Mann, S., Lo, R.C.H., Ovtcharov, K., Gu, S., Dai, D., Ngan, C., Ai, T.: Realtime HDR (high dynamic range) video for eyetap wearable computers, FPGA-based seeing aids, and glasseyes (eyetaps). In: 2012 25th IEEE Canadian Conference on Electrical and Computer Engineering (CCECE) (2012)

  19. Mantiuk, R., Daly, S., Kerofsky, L.: Display adaptive tone mapping. ACM Trans. Graph (TOG), 27, 68 (2008)

    Article  Google Scholar 

  20. Mantiuk, R., Mantiuk, R., Tomaszewska, A., Heidrich, W.: Color correction for tone mapping. In: Computer graphics forum, pp. 193–202. Blackwell Publishing Ltd, Oxford, UK (2009)

    Google Scholar 

  21. Marsi, S., Impoco, G., Ukovich, A., Carrato, S., Ramponi, G.: Video enhancement and dynamic range control of HDR sequences for automotive applications. EURASIP J. Adv. Signal Process. 2007, 080971 (2007)

    Article  Google Scholar 

  22. Mertens, T., Kautz, J., Reeth, F.V.: Exposure fusion. In: Computer Graphics and Applications, 2007. PG ’07. 15th Pacific Conference on (2007)

  23. Min, T.H., Park, R.H., Chang, S.: Histogram based ghost removal in high dynamic range images. In: IEEE International Conference on Multimedia and Expo, 2009. ICME 2009 (2009)

  24. Mitsunaga, T., Nayar, S.K.: Radiometric self calibration. In: Proceedings of 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), vol. 1 (1999)

  25. Ofili, C., Glozman, S.: Yadid-Pecht: hardware implementation of an automatic rendering tone mapping algorithm for a wide dynamic range display. J. Low Power Electron. Appl. 3, 337–367 (2013)

    Article  Google Scholar 

  26. Pece, F., Kautz, J.: Bitmap movement detection: Hdr for dynamic scenes. In: IEEE 2010 Conference on Visual Media Production (CVMP) (2010)

  27. Popadić, I., Todorović, B.M., Reljin, I.: Method for HDR-like imaging using industrial digital cameras. Multimed. Tools Appl. 76, 12801–12817 (2017)

    Article  Google Scholar 

  28. Reinhard, E., Stark, M., Shirley, P., Ferwerda, J.: Photographic tone reproduction for digital images. ACM Trans. Graph. 21, 267–276 (2002)

    Article  Google Scholar 

  29. Robertson, M.A., Borman, S., Stevenson, R.L.: Estimation-theoretic approach to dynamic range enhancement using multiple exposures. J. Electron. Imaging 12, 219–229 (2003)

    Article  Google Scholar 

  30. Sidibe, D., Puech, W., Strauss, O.: Ghost detection and removal in high dynamic range images. In: 2009 17th European Signal Processing Conference (2009)

  31. Tamburrino, D., Alleysson, D., Meylan, L., Süsstrunk, S.: Digital camera workflow for high dynamic range images using a model of retinal processing. In: IST/SPIE Electronic Imaging: Digital Photography IV, vol. 6817 (2008)

  32. Tocci, M.D., Kiser, C., Tocci, N., Sen, P.: A versatile hdr video production system. In: ACM SIGGRAPH 2011 Papers, SIGGRAPH ’11. ACM, New York, NY, USA (2011)

  33. Ureña, R., Martínez-Cañada, P., Gómez-López, J.M., Morillas, C., Pelayo, F.: Real-time tone mapping on GPU and FPGA. EURASIP J. Image Video Process. 1 (2012)

    Article  Google Scholar 

  34. Vytla, L., Hassan, F., Carletta, J.E.: A real-time implementation of gradient domain high dynamic range compression using a local poisson solver. J. Real Time Image Process. 8, 153–167 (2013)

    Article  Google Scholar 

  35. Zhao, H., Shi, B., Fernandez-Cull, C., Yeung, S.K., Raskar, R.: Unbounded high dynamic range photography using a modulo camera. In: ICCP (2015)

Download references

Acknowledgements

This work was supported by The Ministry of Education, Youth and Sports of the Czech Republic from the National Programme of Sustainability (NPU II); project IT4Innovations excellence in science—LQ1602.

Author information

Authors and Affiliations

  1. Brno University of Technology, FIT, Brno, Czech Republic

    Svetozar Nosko, Martin Musil, Pavel Zemcik & Roman Juranek

Authors
  1. Svetozar Nosko
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Martin Musil
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pavel Zemcik
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Roman Juranek
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Svetozar Nosko.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nosko, S., Musil, M., Zemcik, P. et al. Color HDR video processing architecture for smart camera. J Real-Time Image Proc 17, 555–566 (2020). https://doi.org/10.1007/s11554-018-0810-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-018-0810-z

Keywords

Search

Navigation