Skip to main content
SpringerLink
Log in

A flexible mixed-signal image processing pipeline using 3D chip stacks

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

This work presents a highly flexible mixed-signal CMOS image sensor suitable for smart camera applications. These systems need to fit different constraints regarding power consumption, speed and quality, and the optimal compromise may differ depending on the application. Moreover, the best implementation of a desired image processing task may be in the analog or the digital domain, or even a combined computation. Different aspects starting from the image sensor and signal acquisition up to the pre-processing in analog and digital domain are investigated in this paper to optimize not just one part of the system, but the whole system altogether. Moreover, it is shown that analog processing algorithms can improve signal quality, processing speed and latency while being able to save power, which is important for real-time systems. In order to be able to carry out spatial operations, the state-of-the-art sensor is modified to be able to read out multiple pixels at the same time. This allows analog spatial filter operations which consume significantly less power. As an example, an averaging filter is described which needs less than 5.3 % of the power–time product of a digital implementation for one computation. To enhance data throughput and flexibility, 3D chip stacking is proposed to partition the sensor in smaller units and enable massively parallel processing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25

Similar content being viewed by others

References

  1. Allen, P.E., Holberg, D.R.: CMOS Analog Circuit Design. Oxford University Press, Oxford (2002)

    Google Scholar 

  2. Andriani, S., Brendel, H., Seybold, T., Goldstone, J.: Beyond the KODAK image set: a new reference set of color image sequences. In: Proceedings of the IEEE International Conference on Image Processing, Melbourne, Australia (2013)

  3. Cheng, C.-H., Hsieh, H.-C., Fan, T.-Y., Tang, W.-X., Liu, C.-K., Huang, P.-H.: High resolution and frame rate image signal processor array design for 3-D imager. In: International Symposium on Intelligent Signal Processing and Communications Sys- tems (ISPACS), 2012, pp. 735–739 (2012)

  4. Di Federico, M., Julian, P., Andreou, A.G., Mandolesi, P.S.: Fully functional fine-grain vertically integrated 3D focal plane neuromorphic processor. In: SOI-3D-Subthreshold Microelectronics Technology Unified Conference (S3S), 2014, pp. 1–2 (2014)

  5. Dubois, J., Ginhac, D., Paindavoine, M., Heyrman, B.: A 10,000 fps CMOS sensor with massively parallel image processing. Solid-State Circ, IEEE J, 43(3), 706–717 (2008). ISSN: 0018-9200. doi:10.1109/JSSC.2007.916618

  6. Dudek, P., Hicks, P.J.: An analogue SIMD focal-plane processor array. In: Circuits and Systems, 2001. ISCAS 2001. The 2001 IEEE International Symposium on, vol 4, pp. 490–493 (2001). doi:10.1109/ISCAS.2001.922281

  7. Farooque, M.A., Rohankar, J.S.: Survey on various noises and techniques for denoising the color image. In: International Journal of Application or Innovation in Engineering & Management (IJAIEM), vol. 2, no. 11, pp. 217–221 (2013)

  8. Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 3rd edn. Prentice-Hall Inc, Upper Saddle River (2006). ISBN: 013168728X

  9. Goto, M., Hagiwara, K., Iguchi, Y., Ohtake, H., Saraya, T., Kobayashi, M., Higurashi, E., Toshiyoshi, H., Hiramoto, T.: Three-dimensional integrated CMOS image sensors with pixel-parallel A/D converters fabricated by direct bonding of SOI layers. In: International Electron Devices Meeting (IEDM), 2014, pp. 4.2.1–4.2.4 (2014)

  10. Gruev, V., Etienne-Cummings, R.: Implementation of steerable spatiotemporal image filters on the focal plane. Circ. Syst. II: Analog Dig. Signal Process. IEEE Trans., 49(4): 233–244 (2002). ISSN: 1057-7130. doi:10.1109/TCSII.2002.801211

  11. Hartmann C., Reichenbach M., Fey D.: IPOL—a domain specific language for image processing applications. International Conference on Systems, Proceedings of the International Symposium on (2015a). www.thinkmind.org/download.php?articleid=icons_2015_3_20_40047

  12. Hartmann C., Yupatova A., Reichenbach M., Fey D., German R.: A holistic approach for modeling and synthesis of image processing applications for heterogeneous computing architectures. CoRR, abs/1502.07453 (2015b). http://arxiv.org/abs/1502.07453

  13. Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: Pattern Recognition (ICPR), 2010 20th International Conference on, pp. 2366–2369 (2010). doi:10.1109/ICPR.2010.579

  14. Hornsey, R.I., University of Waterloo.: Lecture notes: noise in image sensors (2008)

  15. Lee, C., Chao, W., Lee, S., Hone, J., Molnar, A., Hong, S. H.: A low-power edge detection image sensor based on parallel digital pulse computation. IEEE Trans. Circ. Syst. II: Express Briefs 62(11), 1043–1047 (2015). ISSN: 1549-7747. doi:10.1109/TCSII.2015.2455354

  16. Lopich, A., Dudek, P.: Architecture and design of a programmable 3D-integrated cellular processor array for image processing. In: 19th International Conference onVLSI and System-on-Chip (VLSI-SoC), 2011, pp. 349–353 (2011)

  17. Junichi, N.: Image Sensors and Signal Processing for Digital Still Cameras. CRC PRESS, Boca Raton (2005)

    Google Scholar 

  18. Njuguna, R., Gruev, V.: Low power programmable current mode computational imaging sensor. Sensors J. IEEE 12(4), 727–736 (2012). ISSN: 1530-437X. doi:10.1109/JSEN.2011.2158579

  19. Ohta, J.: Smart CMOS Image Sensors and Applications. CRC Press, Boca Raton (2008)

    Google Scholar 

  20. Pfundt, B., Reichenbach, M., Fey, D., Söll, C.: Novel image processing architecture for 3d integrated circuits. In: Proceedings of the 26th Workshop on Parallel Systems and Algorithms (PARS 2015), Potsdam. To appear (2015)

  21. Rodriguez-Vazquez, A., Carmona-Galan, R., Berni, J. Fernandez, Vargas, S., Lenero, J.A., Suarez, M., Brea, V., Perez-Verdu, B.: Form factor improvement of smart-pixels for vision sensors through 3-D vertically-integrated technologies. In: Circuits and Systems (LASCAS), 2014 IEEE 5th Latin American Symposium on, pp. 1–4 (2014)

  22. Sadhu, B., Sturm, M., Sadler, B.M., Harjani, R.: A 5gs/s 12.2pj/conv. analog charge-domain FFT for a software defined radio receiver front-end in 65nm cmos. In: Radio Frequency Integrated Circuits Symposium (RFIC), 2012 IEEE, pp. 39–42 (2012). doi:10.1109/RFIC.2012.6242227

  23. Schmidt, M., Reichenbach, M., Fey, D.: A generic vhdl template for 2d stencil code applications on fpgas. In: 15th IEEE International Symposium on Object/Component/Service-Oriented Real-Time Distributed Computing Workshops (ISORCW), 2012, pp. 180–187 (2012)

  24. Shi, L., Soell, C., Baenisch, A., Weigel, R., Seiler, J., Ussmueller, T.: Concept for a CMOS image sensor suited for analog image pre-processing. In: Design, Automation and Test in Europe, pp. 16–21 (2015). http://arxiv.org/abs/1502.07449

  25. Soell, C., Shi, L., Baenisch, A., Ussmueller, T., Weigel, R.: A CMOS image sensor with analog pre-processing capability suitable for smart camera applications. In: 2015 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), pp. 279–284 (2015). doi:10.1109/ISPACS.2015.7432780

  26. Suarez, M., Brea, V.M., Pardo, F., Carmona-Galan, R., Rodriguez-Vazquez, A.: A CMOS-3D reconfigurable architecture with in-pixel processing for feature detectors. In: International 3D Systems Integration Conference (3DIC), 2011, pp. 1–8 (2012)

  27. Tietze, U., Schenk, C., Gamm, E.: Electronic Circuits: Handbook for Design and Application. Springer-Verlag GmbH (2008). ISBN: 9783642310256

  28. Yadid-Pecht, O., Etienne-Cummings, R.: CMOS Imagers: From Phototransduction to Image Processing. Kluwer, Berlin (2004)

    Book  Google Scholar 

  29. Zwick, D., Brothers, D. L.: Rms granularity: determination of just-noticeable differences. SMPTE J., 86(6), 427–430 (1977). ISSN: 0036-1682. doi:10.5594/J08125

Download references

Acknowledgments

The authors would like to thank the “Deutsche Forschungsgemeinschaft” (DFG) for funding this project (GRK 1773 “Heterogeneous Image Systems”).

Author information

Authors and Affiliations

  1. Institute for Electronics Engineering, University of Erlangen-Nuremberg, Cauerstr. 9, 91058, Erlangen, Germany

    Lan Shi, Christopher Soell, Andreas Baenisch & Robert Weigel

  2. Chair of Computer Architecture, University of Erlangen-Nuremberg, Martensstr. 3, 91058, Erlangen, Germany

    Benjamin Pfundt & Marc Reichenbach

  3. Chair of Multimedia Communications and Signal Processing, University of Erlangen-Nuremberg, Cauerstr. 7, 91058, Erlangen, Germany

    Juergen Seiler

  4. Institute for Mechatronics, University of Innsbruck, Technikerstrasse, 6020, Innsbruck, Austria

    Thomas Ussmueller

Authors
  1. Lan Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christopher Soell
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Benjamin Pfundt
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andreas Baenisch
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marc Reichenbach
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Juergen Seiler
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Thomas Ussmueller
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Robert Weigel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lan Shi.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shi, L., Soell, C., Pfundt, B. et al. A flexible mixed-signal image processing pipeline using 3D chip stacks. J Real-Time Image Proc 14, 517–534 (2018). https://doi.org/10.1007/s11554-016-0628-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-016-0628-5

Keywords

Search

Navigation