Advertisement

Journal of Real-Time Image Processing

, Volume 12, Issue 4, pp 747–762 | Cite as

HDR-ARtiSt: an adaptive real-time smart camera for high dynamic range imaging

  • Pierre-Jean Lapray
  • Barthélémy Heyrman
  • Dominique Ginhac
Special Issue Paper

Abstract

This paper describes a complete FPGA-based smart camera architecture named HDR-ARtiSt (High Dynamic Range Adaptive Real-time Smart camera) which produces a real-time high dynamic range (HDR) live video stream from multiple captures. A specific memory management unit has been defined to adjust the number of acquisitions to improve HDR quality. This smart camera is built around a standard B&W CMOS image sensor and a Xilinx FPGA. It embeds multiple captures, HDR processing, data display and transfer, which is an original contribution compared to the state-of-the-art. The proposed architecture enables a real-time HDR video flow for a full sensor resolution (1.3 Mega pixels) at 60 frames per second.

Keywords

Smart camera High dynamic range, memory management core Parallel processing FPGA implementation 

Notes

Acknowledgments

This work was supported by the DGCIS (French Ministry for Industry) within the framework of the European HiDRaLoN project and by grants from the Conseil Regional de Bourgogne.

References

  1. 1.
    Acosta-Serafini, P., Ichiro, M., Sodini, C.: A 1/3" VGA linear wide dynamic range CMOS image sensor implementing a predictive multiple sampling algorithm with overlapping integration intervals. IEEE J. Solid-State Circuits 39(9), 1487–1496 (2004)CrossRefGoogle Scholar
  2. 2.
    Akil, M., Grandpierre, T., Perroton, L.: Real-time dynamic tone-mapping operator on GPU. J. Real-Time Image Process. 7(3), 165–172 (2012)CrossRefGoogle Scholar
  3. 3.
    Akyuz, A.: High dynamic range imaging pipeline on the GPU. J. Real-Time Image Process, pp. 1–15. doi: 10.1007/s11554-012-0270-9 (2012)
  4. 4.
    Alston, L., Levinstone, D., Plummer, W.: Exposure control system for an electronic imaging camera having increased dynamic range. US Patent 4647975 A, Cambrige, Mass: U.S. Patent and Trademark Office (1987)Google Scholar
  5. 5.
    Boschetti, A., Adami, N., Leonardi, R., Okuda, M.: An optimal Video-Surveillance approach for HDR videos tone mapping. In: 19th European Signal Processing Conference EUSIPCO 2011, Barcelona (2011)Google Scholar
  6. 6.
    Čadík, M., Wimmer, M., Neumann, L., Artusi, A.: Evaluation of HDR tone mapping methods using essential perceptual attributes. Comput. Graph. 32, 330–349 (2008)CrossRefGoogle Scholar
  7. 7.
    Cembrano, G., Rodriguez-Vazquez, A., Galan, R., Jimenez-Garrido, F., Espejo, S., Dominguez-Castro, R.: A 1000 fps at 128 × 128 vision processor with 8-bit digitized I/O. IEEE J. Solid-State Circuits 39(7), 1044–1055 (2004)CrossRefGoogle Scholar
  8. 8.
    Chiu, C.T., Wang, T.H., Ke, W.M., Chuang, C.Y., Huang, J.S., Wong, W.S., Tsay, R.S., Wu, C.J.: Real-time tone-mapping processor with integrated photographic and gradient compression using 0.13 μm technology on an ARM SoC platform. J. Signal Process. Syst. 64(1), 93–107 (2010)CrossRefGoogle Scholar
  9. 9.
    Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 369–378 (1997)Google Scholar
  10. 10.
    Devlin, K., Chalmers, A., Wilkie, A., Purgathofer, W.: Tone reproduction and physically based spectral rendering. In: Eurographics 2002, Eurographics Association, pp. 101–123 (2002)Google Scholar
  11. 11.
    Duan, J., Bressan, M., Dance, C., Qiu, G.: Tone-mapping high dynamic range images by novel histogram adjustment. Pattern Recognit. 43, 1847–1862 (2010)CrossRefGoogle Scholar
  12. 12.
    Dubois, J., Ginhac, D., Paindavoine, M., Heyrman, B.: A 10 000 fps CMOS sensor with massively parallel image processing. IEEE J. Solid-State Circuits 43(3), 706–717 (2008)CrossRefGoogle Scholar
  13. 13.
    Drago, F., Myszkowski, K., Annen, T., Chiba, N.: Adaptive logarithmic mapping for displaying high contrast scenes. Comput. Graph. Forum 22, 419–426 (2003)Google Scholar
  14. 14.
    Fattal, R., Lischinski, D., Werman, M.: Gradient domain high dynamic range compression. ACM Trans. Graph. (TOG) 21, 249–256 (2002)CrossRefGoogle Scholar
  15. 15.
    Yourganov, G., Stuerzlinger, W.: Acquiring high dynamic range video at video rates. Technical report, Department of Computer Science, York University (2001)Google Scholar
  16. 16.
    Gelfand, N., Adams, A., Park, SH., Pulli, K.: Multi-exposure imaging on mobile devices. In: Proceedings of the International Conference on Multimedia, New York, USA, pp. 823–826 (2010)Google Scholar
  17. 17.
    Graf, H.G., Harendt, C., Engelhardt, T., Scherjon, C., Warkentin, K., Richter, H., Burghartz, J.: High dynamic range CMOS imager technologies for biomedical applications. IEEE J. Solid-State Circuits 44(1), 281–289 (2009)CrossRefGoogle Scholar
  18. 18.
    Granados, M., Ajdin, B., Wand, M., Theobalt, C., Seidel, H., Lensch, H.: Optimal HDR reconstruction with linear digital cameras. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 215–222 (2010)Google Scholar
  19. 19.
    Hassan, F., Carletta, J.: A real-time FPGA-based architecture for a Reinhard-like tone mapping operator. In: Proceedings of the 22nd ACM SIGGRAPH/EUROGRAPHICS Symposium on Graphics Hardware, Aire-la-Ville, Switzerland, pp. 65–71 (2007)Google Scholar
  20. 20.
    Hrabar, S., Corke, P., Bosse, M.: High dynamic range stereo vision for outdoor mobile robotics. In: IEEE International Conference on Robotics and Automation, 2009. ICRA ’09, pp. 430–435 (2009)Google Scholar
  21. 21.
    Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High dynamic range video. Technical report, Interactive Visual Media Group, Microsoft Research, Redmond, WA (2003)Google Scholar
  22. 22.
    Kavusi, S., El Gamal, A.: A quantitative study of high dynamic range image sensor architectures. In: Proceedings of the SPIE Electronic Imaging ’04 Conference, vol. 5301, pp. 264–275 (2004)Google Scholar
  23. 23.
    Ke, W.M., Wang, T.H., Chiu, C.T.: Hardware-efficient virtual high dynamic range image reproduction. In: Proceedings of the 16th IEEE International Conference on Image Processing (ICIP’09), Piscataway, NJ, USA, pp. 2665–2668 (2009)Google Scholar
  24. 24.
    Lee, S.H., Woo, H., Kang, M.G.: Global illumination invariant object detection with level set based bimodal segmentation. IEEE Trans. Circuits Syst. Video Technol. 20(4), 616–620 (2010)CrossRefGoogle Scholar
  25. 25.
    Leflar, M., Hesham, O., Joslin, C.: Use of high dynamic range images for improved medical simulations. In: Magnenat-Thalmann, N. (ed.) Modelling the Physiological Human. Lecture Notes in Computer Science, vol 5903, pp. 199–208. Springer, Berlin (2009)Google Scholar
  26. 26.
    Lindgren, L., Melander, J., Johansson, R., Moller, B.: A multiresolution 100-GOPS 4-Gpixels/s programmable smart vision sensor for multisense imaging. IEEE J. Solid-State Circuits 40(6), 1350–1359 (2005)CrossRefGoogle Scholar
  27. 27.
    Liu, J., Hassan, F., Carletta, J.: A study of hardware-friendly methods for gradient domain tone mapping of high dynamic range images. J. Real-Time Image Process., pp. 1–17. doi: 10.1007/s11554-013-0365-y (2013)
  28. 28.
    Mangiat, S., Gibson, J.: Inexpensive high dynamic range video for large scale security and surveillance. In: Military Communications Conference, 2011—MILCOM 2011, pp. 1772 –1777 (2011)Google Scholar
  29. 29.
    Mitsunaga, T., Nayar, S.: Radiometric self calibration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 374–380 (1999)Google Scholar
  30. 30.
    Morfu, S., Marquié, P., Nofiélé, B., Ginhac, D.: Nonlinear systems for image processing. In: Hawkes, P.W. (ed.) Advances in Imaging and Electron Physics, vol. 152, pp. 79 – 151. Elsevier, Amsterdam (2008)Google Scholar
  31. 31.
    Gallo, O., Gelfand, N., Chen, W., Tico, M., Pulli, K.: Artifact-free high dynamic range imaging. Technical report, University of California (2009)Google Scholar
  32. 32.
    Pattanaik, S.N., Tumblin, J., Yee, H., Greenberg, D.P.: Time-dependent visual adaptation for fast realistic image display. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 47–54 (2000)Google Scholar
  33. 33.
    Reinhard, E., Devlin, K.: Dynamic range reduction inspired by photoreceptor physiology. IEEE Trans. Vis. Comput. Graph. 11, 13–24 (2005)CrossRefGoogle Scholar
  34. 34.
    Reinhard, E., Stark, M., Shirley, P., Ferwerda, J.: Photographic tone reproduction for digital images. ACM Trans. Graph. (TOG) 21(3), 267–276 (2002)CrossRefGoogle Scholar
  35. 35.
    Reinhard, E., Ward, G., Pattanaik, S., Debevec, P., Heidrich, W., Myszkowski, K.: High Dynamic Range Imaging: Acquisition, Display, and Image-based Lighting, 2nd edn. The Morgan Kaufmann Series in Computer Graphics. Elsevier, Burlington (2010)Google Scholar
  36. 36.
    Robertson, M.A., Borman, S., Stevenson, R.L.: Estimation-theoretic approach to dynamic range enhancement using multiple exposures. J. Electron. Imaging 12(2), 219–228 (2003)CrossRefGoogle Scholar
  37. 37.
    Ruedi, P.F., Heim, P., Gyger, S., Kaess, F., Arm, C., Caseiro, R., Nagel, J.L., Todeschini, S.: An SoC combining a 132dB QVGA pixel array and a 32b DSP/MCU processor for vision applications. In: Solid-State Circuits Conference—Digest of Technical Papers, 2009. ISSCC 2009. IEEE International, pp. 46–47,47a (2009)Google Scholar
  38. 38.
    Sakakibara, M., Kawahito, S., Handoko, D., Nakamura, N., Higashi, M., Mabuchi, K., Sumi, H.: A high-sensitivity CMOS image sensor with gain-adaptive column amplifiers. IEEE J. Solid-State Circuits 40(5), 1147–1156 (2005)CrossRefGoogle Scholar
  39. 39.
    Schanz, M., Nitta, C., Bussmann, A., Hosticka, B., Wertheimer, R.: A high-dynamic-range CMOS image sensor for automotive applications. IEEE J. Solid-State Circuits 35(7), 932–938 (2000)CrossRefGoogle Scholar
  40. 40.
    Schlick, C.: Quantization techniques for visualization of high dynamic range pictures. In: Sakas, G., Maller, S., Shirley, P. (eds.): Photorealistic Rendering Techniques, Focus on Computer Graphics. Springer, Berlin, pp. 7–20 (1995)Google Scholar
  41. 41.
    Schubert, F., Schertler, K., Mikolajczyk, K.: A hands-on approach to high-dynamic-range and superresolution fusion. In: IEEE Workshop on Applications of Computer Vision (WACV), pp. 1–8 (2009)Google Scholar
  42. 42.
    Sérot, J., Ginhac, D., Chapuis, R., Dérutin, J.P.: Fast prototyping of parallel-vision applications using functional skeletons. Mach. Vis. Appl. 12, 271–290 (2001)CrossRefzbMATHGoogle Scholar
  43. 43.
    Slomp, M., Oliveira, M.M.: Real-time photographic local tone reproduction using summed-area tables. In: Computer Graphics International, pp. 82–91 (2008)Google Scholar
  44. 44.
    Tamburrino, D., Alleysson, D., Meylan, L., Suesstrunk, S.: Digital camera workflow for high dynamic range images using a model of retinal processing. In: DiCarlo, J., Rodricks, B. (eds.): Digital Photography IV, Proceedings of SPIE, vol 6817, Conference on Digital Photography IV, San Jose, CA, 28–29 Jan 2008 (2008)Google Scholar
  45. 45.
  46. 46.
    Tocci, M.D., Kiser, C., Tocci, N., Sen, P.: A versatile HDR video production system. ACM Trans. Graph. 30(4), 41:1–41:10 (2011)Google Scholar
  47. 47.
    Vytla, L., Hassan, F., Carletta, J.: A real-time implementation of gradient domain high dynamic range compression using a local poisson solver. J. Real-Time Image Process. 8(2), 153–167 (2013)CrossRefGoogle Scholar
  48. 48.
    Wang, Z., Bovik, A.C.: A universal image quality index. IEEE Signal Process. Lett. 9(3), 81–84 (2002)CrossRefGoogle Scholar
  49. 49.
    Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  50. 50.
    Yoshida, A., Blanz, V., Myszkowski, K., Seidel, H.P.: Perceptual evaluation of tone mapping operators with real-world scenes. In: Human Vision and Electronic Imaging X, SPIE, pp. 192–203 (2005)Google Scholar
  51. 51.
    Youm, S.J., Cho, W.H., Hong, K.S.: High dynamic range video through fusion of exposure-controlled frames. In: Proceedings of IAPR Conference on Machine Vision Applications, pp. 546–549 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Pierre-Jean Lapray
    • 1
  • Barthélémy Heyrman
    • 1
  • Dominique Ginhac
    • 1
  1. 1.Laboratory of Electronic, Computing and Imaging Sciences, Le2i UMR 6306University of BurgundyDijonFrance

Personalised recommendations