HDR Imaging Pipeline for Spectral Filter Array Cameras

  • Jean-Baptiste Thomas
  • Pierre-Jean Lapray
  • Pierre Gouton
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10270)

Abstract

Multispectral single shot imaging systems can benefit computer vision applications in needs of a compact and affordable imaging system. Spectral filter arrays technology meets the requirement, but can lead to artifacts due to inhomogeneous intensity levels between spectral channels due to filter manufacturing constraints, illumination and object properties. One solution to solve this problem is to use high dynamic range imaging techniques on these sensors. We define a spectral imaging pipeline that incorporates high dynamic range, demosaicing and color image visualization. Qualitative evaluation is based on real images captured with a prototype of spectral filter array sensor in the visible and near infrared.

Keywords

Multispectral imaging Spectral filter arrays High dynamic range Imaging pipeline 

References

  1. 1.
    An, J., Ha, S.J., Cho, N.I.: Probabilistic motion pixel detection for the reduction of ghost artifacts in high dynamic range images from multiple exposures. EURASIP J. Image Video Process. 2014(1), 42 (2014)CrossRefGoogle Scholar
  2. 2.
    Banterle, F., Artusi, A., Debattista, K., Chalmers, A.: Advanced High Dynamic Range Imaging: Theory and Practice. AK Peters (CRC Press), Natick (2011)CrossRefGoogle Scholar
  3. 3.
    Bouderbane, M., Lapray, P.J., Dubois, J., Heyrman, B., Ginhac, D.: Real-time ghost free HDR video stream generation using weight adaptation based method. In: Proceedings of the 10th International Conference on Distributed Smart Camera, ICDSC 2016, pp. 116–120. ACM, New York (2016)Google Scholar
  4. 4.
    Brauers, J., Schulte, N., Bell, A., Aach, T.: Color accuracy and noise analysis in multispectral HDR imaging. In: 14. Workshop Farbbildverarbeitung 2008, pp. 33–42. Shaker Verlag (2008)Google Scholar
  5. 5.
    Brochard, N., Nebhen, J., Ginhac, D.: 3D-IC: new perspectives for a digital pixel sensor. In: Proceedings of the 10th International Conference on Distributed Smart Camera, ICDSC 2016, pp. 92–97. ACM, New York (2016)Google Scholar
  6. 6.
    Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1997, pp. 369–378. ACM Press/Addison-Wesley Publishing Co., New York (1997)Google Scholar
  7. 7.
    Duan, J., Bressan, M., Dance, C., Qiu, G.: Tone-mapping high dynamic range images by novel histogram adjustment. Pattern Recogn. 43, 1847–1862 (2010)CrossRefGoogle Scholar
  8. 8.
    Eichenholz, J.M., Dougherty, J.: Ultracompact fully integrated megapixel multispectral imager. In: SPIE OPTO: Integrated Optoelectronic Devices, p. 721814. International Society for Optics and Photonics (2009)Google Scholar
  9. 9.
    Geelen, B., Blanch, C., Gonzalez, P., Tack, N., Lambrechts, A.: A tiny VIS-NIR snapshot multispectral camera. In: SPIE OPTO, p. 937414. International Society for Optics and Photonics (2015)Google Scholar
  10. 10.
    Granados, M., Kim, K.I., Tompkin, J., Theobalt, C.: Automatic noise modeling for ghost-free HDR reconstruction. ACM Trans. Graph. (TOG) 32(6), 201 (2013)CrossRefGoogle Scholar
  11. 11.
    Hanhart, P., Bernardo, M.V., Pereira, M., Pinheiro, A.M.G., Ebrahimi, T.: Benchmarking of objective quality metrics for HDR image quality assessment. EURASIP J. Image Video Process. 2015(1), 39 (2015)CrossRefGoogle Scholar
  12. 12.
    IMEC: hyperspectral-imaging. http://www2.imec.be
  13. 13.
    Jia, J., Barnard, K.J., Hirakawa, K.: Fourier spectral filter array for optimal multispectral imaging. IEEE Trans. Image Process. 25(4), 1530–1543 (2016)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Kao, W.C., Wang, S.H., Chen, L.Y., Lin, S.Y.: Design considerations of color image processing pipeline for digital cameras. IEEE Trans. Consum. Electron. 52(4), 1144–1152 (2006)CrossRefGoogle Scholar
  15. 15.
    Kiku, D., Monno, Y., Tanaka, M., Okutomi, M.: Simultaneous capturing of RGB and additional band images using hybrid color filter array. In: Proceedings of the SPIE, vol. 9023, pp. 90230V–90230V-9 (2014)Google Scholar
  16. 16.
    Knickerbocker, J.U., Andry, P., Dang, B., Horton, R., Patel, C.S., Polastre, R., Sakuma, K., Sprogis, E., Tsang, C., Webb, B., et al.: 3D silicon integration. In: 2008 58th Electronic Components and Technology Conference, pp. 538–543. IEEE (2008)Google Scholar
  17. 17.
    Krawczyk, G., Myszkowski, K., Seidel, H.P.: Lightness perception in tone reproduction for high dynamic range images. Comput. Graph. Forum 24, 635–645 (2005). Wiley Online LibraryCrossRefGoogle Scholar
  18. 18.
    Lapray, P.J., Heyrman, B., Ginhac, D.: HDR-ARtiSt: an adaptive real-time smart camera for high dynamic range imaging. J. Real-Time Image Process., 1–16 (2014)Google Scholar
  19. 19.
    Lapray, P.J., Heyrman, B., Ginhac, D.: Hardware-based smart camera for recovering high dynamic range video from multiple exposures. Opt. Eng. 53(10), 102–110 (2014)CrossRefGoogle Scholar
  20. 20.
    Lapray, P.J., Thomas, J.B., Gouton, P.: A multispectral acquisition system using MSFAs. In: Color and Imaging Conference, vol. 2014(2014), pp. 97–102 (2014–11-03T00: 00: 00)Google Scholar
  21. 21.
    Lapray, P.J., Thomas, J.B., Gouton, P., Ruichek, Y.: Energy balance in spectral filter array camera design. J. Eur. Opt. Soc. (2017)Google Scholar
  22. 22.
    Lapray, P.J., Wang, X., Thomas, J.B., Gouton, P.: Multispectral filter arrays: recent advances and practical implementation. Sensors 14(11), 21626 (2014)CrossRefGoogle Scholar
  23. 23.
    Larson, G.W., Shakespeare, R.: Rendering with Radiance: The Art and Science of Lighting Visualization. Booksurge LLC, New York (2004)Google Scholar
  24. 24.
    Mann, S., Picard, R.: Being undigital with digital cameras: extending dynamic range by combining differently exposed pictures. In: Proceedings of IS&T 46th Annual Conference, pp. 422–428 (1995)Google Scholar
  25. 25.
    Mann, S., Lo, R.C.H., Ovtcharov, K., Gu, S., Dai, D., Ngan, C., Ai, T.: Realtime HDR (high dynamic range) video for eyetap wearable computers, FPGA-based seeing aids, and glasseyes (eyetaps). In: 2012 25th IEEE Canadian Conference on Electrical & Computer Engineering (CCECE), pp. 1–6. IEEE (2012)Google Scholar
  26. 26.
    Miao, L., Qi, H., Ramanath, R., Snyder, W.E.: Binary tree-based generic demosaicking algorithm for multispectral filter arrays. IEEE Trans. Image Process. 15(11), 3550–3558 (2006)CrossRefGoogle Scholar
  27. 27.
    Mitsunaga, T., Nayar, S.K.: Radiometric self calibration. In: 1999 IEEE Computer Society Conference on CVPR, vol. 1, p. 380 (1999)Google Scholar
  28. 28.
    Monno, Y., Kikuchi, S., Tanaka, M., Okutomi, M.: A practical one-shot multispectral imaging system using a single image sensor. IEEE Trans. Image Process. 24(10), 3048–3059 (2015)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Park, H., Dan, Y., Seo, K., Yu, Y.J., Duane, P.K., Wober, M., Crozier, K.B.: Vertical silicon nanowire photodetectors: Spectral sensitivity via nanowire radius. In: CLEO: Science and Innovations, p. CTh3L-5. OSA (2013)Google Scholar
  30. 30.
    Péguillet, H., Thomas, J.B., Gouton, P., Ruichek, Y.: Energy balance in single exposure multispectral sensors. In: CVCS 2013, pp. 1–6, September 2013Google Scholar
  31. 31.
    PIXELTEQ: Micro-patterned optical filters. https://pixelteq.com/
  32. 32.
    Ramanath, R., Snyder, W.E., Yoo, Y., Drew, M.S.: Color image processing pipeline. IEEE Signal Process. Mag. 22(1), 34–43 (2005)CrossRefGoogle Scholar
  33. 33.
    Rani, K.S., Hans, W.J.: FPGA implementation of bilinear interpolation algorithm for CFA demosaicing. In: 2013 International Conference on Communications and Signal Processing (ICCSP), pp. 857–863. IEEE (2013)Google Scholar
  34. 34.
    Reinhard, E., Stark, M., Shirley, P., Ferwerda, J.: Photographic tone reproduction for digital images. ACM Trans. Graph. 21(3), 267–276 (2002)CrossRefGoogle Scholar
  35. 35.
    Robertson, M.A., Borman, S., Stevenson, R.L.: Dynamic range improvement through multiple exposures. In: Proceedings of the 1999 International Conference on Image Processing, ICIP 1999, vol. 3, pp. 159–163. IEEE (1999)Google Scholar
  36. 36.
    Sadeghipoor, Z., Lu, Y.M., Mendez, E., Ssstrunk, S.: Multiscale guided deblurring: chromatic aberration correction in color and near-infrared imaging. In: 2015 23rd European Signal Processing Conference (EUSIPCO), pp. 2336–2340, August 2015Google Scholar
  37. 37.
    Sadeghipoor, Z., Thomas, J.B., Susstrunk, S.: Demultiplexing visible and near-infrared information in single-sensor multispectral imaging. In: Color and Imaging Conference 2016, pp. 76–81 (2016–11-03T00: 00: 00). http://ist.publisher.ingentaconnect.com/contentone/ist/cic/2016/00002016/00000001/art00012
  38. 38.
    Sharma, G., Trussell, H.J.: Digital color imaging. IEEE Trans. Image Process. 6(7), 901–932 (1997)CrossRefGoogle Scholar
  39. 39.
    Simon, P.M.: Single shot high dynamic range and multispectral imaging based on properties of color filter arrays. Ph.D. thesis, University of Dayton (2011)Google Scholar
  40. 40.
    Tamburrino, D., Alleysson, D., Meylan, L., Süsstrunk, S.: Digital camera workflow for high dynamic range images using a model of retinal processing. In: Electronic Imaging 2008, p. 68170J. International Society for Optics and Photonics (2008)Google Scholar
  41. 41.
    Technologies, S.: Micro-optics supplier. http://www.silios.com/
  42. 42.
    Thomas, J.B.: Illuminant estimation from uncalibrated multispectral images. In: CVCS 2015, pp. 1–6. IEEE (2015)Google Scholar
  43. 43.
    Thomas, J.B., Lapray, P.J., Gouton, P., Clerc, C.: Spectral characterization of a prototype SFA camera for joint visible and NIR acquisition. Sensors 16(7), 993 (2016)CrossRefGoogle Scholar
  44. 44.
    Tsin, Y., Ramesh, V., Kanade, T.: Statistical calibration of CCD imaging process. In: Proceedings of the Eighth IEEE International Conference on Computer Vision, ICCV 2001, vol. 1, pp. 480–487. IEEE (2001)Google Scholar
  45. 45.
    Yi, D., Kong, L., Wang, J., Zhao, F.: Fabrication of multispectral imaging technology driven MEMS-based micro-arrayed multichannel optical filter mosaic. In: SPIE MOEMS-MEMS, p. 792711. International Society for Optics and Photonics (2011)Google Scholar
  46. 46.
    Zhou, J.: Getting the most out of your image-processing pipeline. White Paper, Texas Instruments (2007)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Jean-Baptiste Thomas
    • 1
    • 2
  • Pierre-Jean Lapray
    • 3
  • Pierre Gouton
    • 1
  1. 1.Le2i, FRE CNRS 2005Université de Bourgogne, Franche-ComtéDijonFrance
  2. 2.The Norwegian Colour and Visual Computing LaboratoryNTNU - Norwegian University of Science and TechnologyGjøvikNorway
  3. 3.MIPS LaboratoryUniversité de Haute AlsaceMulhouseFrance

Personalised recommendations