Skip to main content
SpringerLink
Log in

Color Pixel Reconstruction for a Monolithic RGB-Z CMOS Imager

  • Published:
Journal of Signal Processing Systems Aims and scope Submit manuscript

Abstract

In this paper, we introduce the challenges, intrinsic and extrinsic, of color and depth sensors integration in the same matrix for a monolithic RBG-Z CMOS imager system. Due to the fact that the technology to conceive this type of circuit is still under development, the challenge that we address is the extrinsic one. It is a consequence of the heterogeneity of the matrix, where information is missing compared to what can be provided by separate RGB and Z systems. For that a first evaluation is done taking into account how the RGB-Z patterns could impact the demosaicing step. The evaluated pattern are in function of the different sizes between color and depth pixels. For the missing color reconstruction we have evaluated the state of the art algorithms, adapted to the missing information, and we propose an original adaptive algorithm using a new operator called semi-gradient (SG). To fill the lack of a mature technology for which real images are missing for this type of CMOS imager, a test environment was created and then used with three different databases, Kodak, McMaster, HDR+burst. The results show improvements on edges, corners, and narrow lines reconstruction, and a reduction of color and structural artefacts compared to the state-of-the-art reconstruction algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Figure 18
Figure 19
Figure 20
Figure 21
Figure 22
Figure 23
Figure 24
Figure 25

Similar content being viewed by others

Notes

  1. Internet of Things

  2. Color Depth Filter Array

  3. Edge Directed Interpolation

  4. Near InfraRed

  5. Peak signal-to-noise-ratio

  6. Color Peak signal-to-noise-ratio

  7. Structural Similarity index

  8. Multi scale Structural Similarity index

  9. stands for International Commission on Illumination. It is the international authority on light, illumination, color, and color spaces

References

  1. Alper, G. (2015). 1000 Frames per second cameras: fast frame rates with ultra high resolution industrial cameras. https://www.adimec.com/1000-frames-per-second-cameras-fast-frame-rates-with-ultra-high-resolution-industrial-cameras/. Accessed 20 Dec. 2021.

  2. Gamal, A. E. (2002). High Dynamic Range Image Sensors. International Solid-State Circuits Conference, 290.

  3. Bevilacqua, A., Di Stefano, L., & Azzari, P. (2006). People tracking using a time-of-flight depth sensor. Proceedings - IEEE International Conference on Video and Signal Based Surveillance. https://doi.org/10.1109/AVSS.2006.92

    Article  Google Scholar 

  4. Microsoft: Microsoft Kinect. https://en.wikipedia.org/wiki/Kinect. Accessed 20 Dec. 2021.

  5. Litomisky, K. (2012). Consumer RGB-d cameras and their applications. Tech. rep. http://alumni.cs.ucr.edu/~klitomis/files/RGBD-intro.pdf. Accessed 20 Dec. 2021.

  6. Tillman, M. (2020). What is Apple Face ID and how does it work? https://www.pocket-lint.com/phones/news/apple/142207-what-is-apple-face-id-and-how-does-it-work. Accessed 20 Dec. 2021.

  7. Gove, R. J. (2020). CMOS image sensor technology advances for mobile devices. In: High Performance Silicon Imaging. Elsevier Ltd., (2nd ed., pp. 185–240). https://doi.org/10.1016/B978-0-08-102434-8.00007-6. http://dx.doi.org/10.1016/B978-0-08-102434-8.00007-6

  8. Kim, W., Yibing, W., Ovsiannikov, I., Lee, S., Park, Y., Chung, C., & Fossum, E. (2012). A 1.5Mpixel RGBZ CMOS image sensor for simultaneous color and range image capture. Digest of Technical Papers - IEEE International Solid-State Circuits Conference, 55, 392–393. https://doi.org/10.1109/ISSCC.2012.6177061

  9. Rodrigues, B., Guillon, M., Billon-Pierron, N., Mancini, J. B., Giffard, B., Cazaux, Y., Malinge, P., Waltz, P., Ngoua, A., Taluy, A., Kuster, S., Joblot, S., Roy, F., Lu, G. N., Lyon, N. D., & Lyon, B. (2017). Indirect ToF Pixel integrating fast buried-channel transfer gates and gradual epitaxy, and enabling CDS. International Image Sensor Workshop, 266–269.

  10. Monno, Y., Teranaka, H., Yoshizaki, K., Tanaka, M., & Okutomi, M. (2019). Single-sensor RGB-NIR imaging: High-quality system design and prototype implementation. IEEE Sensors Journal, 19(2), 497–507. https://doi.org/10.1109/JSEN.2018.2876774

    Article  Google Scholar 

  11. Teranaka, H., Monno, Y., Tanaka, M., & Ok, M. (2016). Single-sensor RGB and NIR image acquisition: toward optimal performance by taking account of CFA Pattern, demosaicking, and color correction. Electronic Imaging, 2016(18), 1–6. https://doi.org/10.2352/ISSN.2470-1173.2016.18.DPMI-256

    Article  Google Scholar 

  12. Navinprashath, R. R., & Radhesh, B. (2019). Learning based demosaicing and color correction for RGB-IR patterned image sensors. Society for Imaging Science and Technology, 1–6. https://doi.org/10.2352/ISSN.2470-1173.2019.15.AVM-045

  13. Koyama, S., Inaba, Y., Kasano, M., & Murata, T. (2008). A day and night vision MOS Imager with robust. 55(3), 754–759.

  14. Shi, L., Ovsiannikov, I., Min, D. K., Noh, Y., Kim, W., & Jung, S. (2013). Demosaicing for RGBZ Sensor. SPIE Proceedings, 8657, 1–9. https://doi.org/10.1117/12.2001702

  15. Shi, L., & Ovsiannikov, I. (2015).  Demosaicing for RGBZ sensor (Nb US 2015/0022869 A1).

  16. Ramanath, R., & Snyder, W. E. (2003). Adaptive demosaicking. Journal of Electronic Imaging, 12, 633–642. https://doi.org/10.1117/1.1606459

    Article  Google Scholar 

  17. Malvar, H., Li-Wei He, & Cutler, R. (2004). High-quality linear interpolation for demosaicing of Bayer-patterned color images. 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, 3, III–485–8. https://doi.org/10.1109/ICASSP.2004.1326587. http://ieeexplore.ieee.org/document/1326587/

  18. Rebiere, V., Drouot, A., Granado, B., Bourge, A., & Pinna, A. (2020). Semi-Gradient for Color Pixel Reconstruction in a RGBZ CMOS Sensor. 2020 IEEE Sensors, 6–9. https://doi.org/10.1109/SENSORS47125.2020.9278887

  19. Breitbarth, A., Schardt, T., Kind, C., Brinkmann, J., Dittrich, P. G., & Notni, G. (2019). Measurement accuracy and dependence on external influences of the iPhone X TrueDepth sensor (October), 7. https://doi.org/10.1117/12.2530544

  20. Alacoque, L. (2015). Method for demosaicing a raw digital image, corresponding computer program and imaging or graphic circuit (number EP 2426639 B1).

  21. Tomasi, C., & Manduchi, R. (1998). Bilateral Filtering for Gray and Color Images. International Conference on Computer Vision.

  22. Horé, A., & Ziou, D. (2011). An edge-sensing generic demosaicing algorithm with application to image resampling. IEEE Transactions on Image Processing, 20(11), 3136–3150. https://doi.org/10.1109/TIP.2011.2159229

    Article  MathSciNet  MATH  Google Scholar 

  23. Adlakha, D., Adlakha, D., & Tanwar, R. (2016). Analytical Comparison between Sobel and Prewitt Edge Detection Techniques. 7(1), 1482–1485.

  24. Alakarhu, J. (2007). Image Sensors and Image Quality in Mobile Phones. International Image Sensor Workshop, Ogunquit (pp. 7–18).

  25. Kodak color image dataset. http://r0k.us/graphics/kodak/. Accessed 20 Dec. 2021.

  26. Zhang, L., Wu, X., Buades, A., & Li, X. (2011). Color Demosaicking by Local Directional Interpolation and Nonlocal Adaptive Thresholding. Electronic Imaging, 1–29.

  27. Hasinoff, S. W., Sharlet, D. Geiss, R., Levoy, A. A., Barron, J. T., Kainz, F., Chen, J., & L., M. (2016). Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics, 35. http://hdrplusdata.org/dataset.html. Accessed 20 Dec. 2021.

  28. Wang, Z., Bovik, A. C., Sheikh, H. R., & Simmoncelli, E. P. (2004). Image quality assessment: form error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612. https://doi.org/10.1109/TIP.2003.819861

  29. Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2003). Multi-scale Structural Similarity For image Quality Assessment. IEEE Asilomar Conference on Signals, Systems and Computer, 2, 9–13.

    Google Scholar 

  30. Lu, W., & Tan, Y. P. (2003). Color filter array demosaicking: new method and performance measures. IEEE Transactions on Image Processing : a publication of the IEEE Signal Processing Society, 12(10), 1194–1210. https://doi.org/10.1109/TIP.2003.816004

    Article  Google Scholar 

  31. Fairchild, M. D. (1997). Color Appearance Models.

  32. Mahy, M., Eycken, L. V., & Oosterlinck, A. (1994). Evaluation of Uniform Color Spaces Developed after the Adoption of CIELAB and CIELUV. Color Research and Application, 19(2), 105–121.

    Google Scholar 

  33. Yasuma, F., Mitsunaga, T., Iso, D., & Nayar, S. K. (2008). Generalized Assorted Pixel Camera: Post-Capture Control of Resolution, Dynamic Range and Spectrum. Tech. rep.

Download references

Author information

Authors and Affiliations

  1. STMicroelectronics, 29 Boulevard Romain Rolland, 92120, Montrouge, France

    Valentin Rebiere, Antoine Drouot & Arnaud Bourge

  2. Sorbonne Université, CNRS, Laboratoire d’Informatique Paris 6 - LIP6, 4, Place Jusssieu, 75005, Paris, France

    Bertrand Granado & Andrea Pinna

Authors
  1. Valentin Rebiere
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antoine Drouot
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bertrand Granado
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Arnaud Bourge
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andrea Pinna
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Valentin Rebiere.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rebiere, V., Drouot, A., Granado, B. et al. Color Pixel Reconstruction for a Monolithic RGB-Z CMOS Imager. J Sign Process Syst 94, 623–644 (2022). https://doi.org/10.1007/s11265-021-01726-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11265-021-01726-3

Keywords

Search

Navigation