Journal of Real-Time Image Processing

, Volume 14, Issue 3, pp 517–534 | Cite as

A flexible mixed-signal image processing pipeline using 3D chip stacks

  • Lan ShiEmail author
  • Christopher Soell
  • Benjamin Pfundt
  • Andreas Baenisch
  • Marc Reichenbach
  • Juergen Seiler
  • Thomas Ussmueller
  • Robert Weigel
Special Issue Paper


This work presents a highly flexible mixed-signal CMOS image sensor suitable for smart camera applications. These systems need to fit different constraints regarding power consumption, speed and quality, and the optimal compromise may differ depending on the application. Moreover, the best implementation of a desired image processing task may be in the analog or the digital domain, or even a combined computation. Different aspects starting from the image sensor and signal acquisition up to the pre-processing in analog and digital domain are investigated in this paper to optimize not just one part of the system, but the whole system altogether. Moreover, it is shown that analog processing algorithms can improve signal quality, processing speed and latency while being able to save power, which is important for real-time systems. In order to be able to carry out spatial operations, the state-of-the-art sensor is modified to be able to read out multiple pixels at the same time. This allows analog spatial filter operations which consume significantly less power. As an example, an averaging filter is described which needs less than 5.3 % of the power–time product of a digital implementation for one computation. To enhance data throughput and flexibility, 3D chip stacking is proposed to partition the sensor in smaller units and enable massively parallel processing.


Smart camera Analog pre-processing Application optimized image pipeline 3D chip stacking Partitioning 



The authors would like to thank the “Deutsche Forschungsgemeinschaft” (DFG) for funding this project (GRK 1773 “Heterogeneous Image Systems”).


  1. 1.
    Allen, P.E., Holberg, D.R.: CMOS Analog Circuit Design. Oxford University Press, Oxford (2002)Google Scholar
  2. 2.
    Andriani, S., Brendel, H., Seybold, T., Goldstone, J.: Beyond the KODAK image set: a new reference set of color image sequences. In: Proceedings of the IEEE International Conference on Image Processing, Melbourne, Australia (2013)Google Scholar
  3. 3.
    Cheng, C.-H., Hsieh, H.-C., Fan, T.-Y., Tang, W.-X., Liu, C.-K., Huang, P.-H.: High resolution and frame rate image signal processor array design for 3-D imager. In: International Symposium on Intelligent Signal Processing and Communications Sys- tems (ISPACS), 2012, pp. 735–739 (2012)Google Scholar
  4. 4.
    Di Federico, M., Julian, P., Andreou, A.G., Mandolesi, P.S.: Fully functional fine-grain vertically integrated 3D focal plane neuromorphic processor. In: SOI-3D-Subthreshold Microelectronics Technology Unified Conference (S3S), 2014, pp. 1–2 (2014)Google Scholar
  5. 5.
    Dubois, J., Ginhac, D., Paindavoine, M., Heyrman, B.: A 10,000 fps CMOS sensor with massively parallel image processing. Solid-State Circ, IEEE J, 43(3), 706–717 (2008). ISSN: 0018-9200. doi: 10.1109/JSSC.2007.916618
  6. 6.
    Dudek, P., Hicks, P.J.: An analogue SIMD focal-plane processor array. In: Circuits and Systems, 2001. ISCAS 2001. The 2001 IEEE International Symposium on, vol 4, pp. 490–493 (2001). doi: 10.1109/ISCAS.2001.922281
  7. 7.
    Farooque, M.A., Rohankar, J.S.: Survey on various noises and techniques for denoising the color image. In: International Journal of Application or Innovation in Engineering & Management (IJAIEM), vol. 2, no. 11, pp. 217–221 (2013)Google Scholar
  8. 8.
    Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 3rd edn. Prentice-Hall Inc, Upper Saddle River (2006). ISBN: 013168728XGoogle Scholar
  9. 9.
    Goto, M., Hagiwara, K., Iguchi, Y., Ohtake, H., Saraya, T., Kobayashi, M., Higurashi, E., Toshiyoshi, H., Hiramoto, T.: Three-dimensional integrated CMOS image sensors with pixel-parallel A/D converters fabricated by direct bonding of SOI layers. In: International Electron Devices Meeting (IEDM), 2014, pp. 4.2.1–4.2.4 (2014)Google Scholar
  10. 10.
    Gruev, V., Etienne-Cummings, R.: Implementation of steerable spatiotemporal image filters on the focal plane. Circ. Syst. II: Analog Dig. Signal Process. IEEE Trans., 49(4): 233–244 (2002). ISSN: 1057-7130. doi: 10.1109/TCSII.2002.801211
  11. 11.
    Hartmann C., Reichenbach M., Fey D.: IPOL—a domain specific language for image processing applications. International Conference on Systems, Proceedings of the International Symposium on (2015a).
  12. 12.
    Hartmann C., Yupatova A., Reichenbach M., Fey D., German R.: A holistic approach for modeling and synthesis of image processing applications for heterogeneous computing architectures. CoRR, abs/1502.07453 (2015b).
  13. 13.
    Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: Pattern Recognition (ICPR), 2010 20th International Conference on, pp. 2366–2369 (2010). doi: 10.1109/ICPR.2010.579
  14. 14.
    Hornsey, R.I., University of Waterloo.: Lecture notes: noise in image sensors (2008)Google Scholar
  15. 15.
    Lee, C., Chao, W., Lee, S., Hone, J., Molnar, A., Hong, S. H.: A low-power edge detection image sensor based on parallel digital pulse computation. IEEE Trans. Circ. Syst. II: Express Briefs 62(11), 1043–1047 (2015). ISSN: 1549-7747. doi: 10.1109/TCSII.2015.2455354
  16. 16.
    Lopich, A., Dudek, P.: Architecture and design of a programmable 3D-integrated cellular processor array for image processing. In: 19th International Conference onVLSI and System-on-Chip (VLSI-SoC), 2011, pp. 349–353 (2011)Google Scholar
  17. 17.
    Junichi, N.: Image Sensors and Signal Processing for Digital Still Cameras. CRC PRESS, Boca Raton (2005)Google Scholar
  18. 18.
    Njuguna, R., Gruev, V.: Low power programmable current mode computational imaging sensor. Sensors J. IEEE 12(4), 727–736 (2012). ISSN: 1530-437X. doi: 10.1109/JSEN.2011.2158579
  19. 19.
    Ohta, J.: Smart CMOS Image Sensors and Applications. CRC Press, Boca Raton (2008)Google Scholar
  20. 20.
    Pfundt, B., Reichenbach, M., Fey, D., Söll, C.: Novel image processing architecture for 3d integrated circuits. In: Proceedings of the 26th Workshop on Parallel Systems and Algorithms (PARS 2015), Potsdam. To appear (2015)Google Scholar
  21. 21.
    Rodriguez-Vazquez, A., Carmona-Galan, R., Berni, J. Fernandez, Vargas, S., Lenero, J.A., Suarez, M., Brea, V., Perez-Verdu, B.: Form factor improvement of smart-pixels for vision sensors through 3-D vertically-integrated technologies. In: Circuits and Systems (LASCAS), 2014 IEEE 5th Latin American Symposium on, pp. 1–4 (2014)Google Scholar
  22. 22.
    Sadhu, B., Sturm, M., Sadler, B.M., Harjani, R.: A 5gs/s 12.2pj/conv. analog charge-domain FFT for a software defined radio receiver front-end in 65nm cmos. In: Radio Frequency Integrated Circuits Symposium (RFIC), 2012 IEEE, pp. 39–42 (2012). doi: 10.1109/RFIC.2012.6242227
  23. 23.
    Schmidt, M., Reichenbach, M., Fey, D.: A generic vhdl template for 2d stencil code applications on fpgas. In: 15th IEEE International Symposium on Object/Component/Service-Oriented Real-Time Distributed Computing Workshops (ISORCW), 2012, pp. 180–187 (2012)Google Scholar
  24. 24.
    Shi, L., Soell, C., Baenisch, A., Weigel, R., Seiler, J., Ussmueller, T.: Concept for a CMOS image sensor suited for analog image pre-processing. In: Design, Automation and Test in Europe, pp. 16–21 (2015).
  25. 25.
    Soell, C., Shi, L., Baenisch, A., Ussmueller, T., Weigel, R.: A CMOS image sensor with analog pre-processing capability suitable for smart camera applications. In: 2015 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), pp. 279–284 (2015). doi: 10.1109/ISPACS.2015.7432780
  26. 26.
    Suarez, M., Brea, V.M., Pardo, F., Carmona-Galan, R., Rodriguez-Vazquez, A.: A CMOS-3D reconfigurable architecture with in-pixel processing for feature detectors. In: International 3D Systems Integration Conference (3DIC), 2011, pp. 1–8 (2012)Google Scholar
  27. 27.
    Tietze, U., Schenk, C., Gamm, E.: Electronic Circuits: Handbook for Design and Application. Springer-Verlag GmbH (2008). ISBN: 9783642310256Google Scholar
  28. 28.
    Yadid-Pecht, O., Etienne-Cummings, R.: CMOS Imagers: From Phototransduction to Image Processing. Kluwer, Berlin (2004)CrossRefGoogle Scholar
  29. 29.
    Zwick, D., Brothers, D. L.: Rms granularity: determination of just-noticeable differences. SMPTE J., 86(6), 427–430 (1977). ISSN: 0036-1682. doi: 10.5594/J08125

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Lan Shi
    • 1
    Email author
  • Christopher Soell
    • 1
  • Benjamin Pfundt
    • 2
  • Andreas Baenisch
    • 1
  • Marc Reichenbach
    • 2
  • Juergen Seiler
    • 3
  • Thomas Ussmueller
    • 4
  • Robert Weigel
    • 1
  1. 1.Institute for Electronics EngineeringUniversity of Erlangen-NurembergErlangenGermany
  2. 2.Chair of Computer ArchitectureUniversity of Erlangen-NurembergErlangenGermany
  3. 3.Chair of Multimedia Communications and Signal ProcessingUniversity of Erlangen-NurembergErlangenGermany
  4. 4.Institute for MechatronicsUniversity of InnsbruckTechnikerstrasseAustria

Personalised recommendations