Abstract
This chapter considers the design of cameras and all the processes that are required to capture perform the initial processing an image. We will concentrate in this chapter on algorithms that provide traditional photos, such as sharpening and compression. Imaging chain algorithms must be designed for efficiency. We measure efficiency along several axes:
-
Execution time. Cameras—both still and video—are real-time systems. We care about the rate at which we can capture, process, and store images. Algorithms must be designed to run fast. We are also concerned about variations in their execution time, which can require additional buffer memory that imposes other costs and limitations.
-
Energy and power consumption. Energy and power are related but distinct concerns. Energy is important because most cameras are battery-powered; lower energy per consumption per image results in more images per battery charge. Energy-efficient algorithms and systems must avoid unnecessary or duplicative work. Power consumption—energy per unit time—is important in large part because of thermal requirements. Power consumption results in heat. Thermal power dissipation is the primary limitation on performance in high-performance computer systems [Wol17]. Heat generated in a camera can also affect sensor performance—most device and circuit noise increases with temperature, typically exponentially.
-
Memory bandwidth and capacity. Multimedia algorithms are memory-intensive. Memory and mass storage devices can absorb and produce data at limited rates. High memory access rates can limit system performance; it can also drive up energy and power consumption. We are also concerned with the total memory usage of an algorithm. Certain parts of the imaging pipeline, particularly those near the image sensor, provide only constrained amounts of memory. Sloppy use of buffer memory can, for example, limit the number of images in a burst.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
N. Ahmed, T. Natarajan and K. R. Rao, “Discrete Cosine Transform,” in IEEE Transactions on Computers, vol. C-23, no. 1, pp. 90–93, Jan 1974. doi: https://doi.org/10.1109/T-C.1974.223784
Aldus and Microsoft, TIFF Revision 5.0, Aug. 8, 1988.
ARM, Cortex-A9 MPCore Technical Reference Manual, Revision r2p0, 2009.
Benjamin B, “Cutting-edge clarity,” in American Cinematographer, vol. 97, no. 12, Dec 2016, pp. 34–49.
Bryce E. Bayer, “Color imaging array,” U. S. Patent 3,971,065, Mar 5, 1975.
Cynthia S. Bell, “Contrast-based autofocus mechanism,” U. S. Patent 5,170,202, Dec 8, 1992.
Cynthia S. Bell, Edward P. Tomaszewski, Amy E. Hansen, and Kannan Raj, “Determining a final exposure setting automatically for a solid state camera without a separate light metering circuit,” U. S. Patent 6,486, 915, Nov 26, 2002.
A. C. Bovik, “Automatic Prediction of Perceptual Image and Video Quality,” in Proceedings of the IEEE, vol. 101, no. 9, Sept 2013, pp. 2008–2024.
W. S. Boyle and G. E. Smith, “Charge Coupled Semiconductor Devices,” in Bell System Technical Journal, vol. 49, no. 4, Apr 1970, pp. 587–593.
Richard Butler, “Exclusive: Fujifilm’s phase detection system explained,” dpreview.com, Aug 5, 2010.
N. Ng Kuang Chern, Poo Aun Neow and M. H. Ang, “Practical issues in pixel-based autofocusing for machine vision,” in Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164), 2001, pp. 2791–2796, vol. 3. doi: https://doi.org/10.1109/ROBOT.2001.933045
Nam Ik Cho and San Uk Lee, “Fast algorithm and implementation of 2-D discrete cosine transform,” in IEEE Transactions on Circuits and Systems, vol. 38, no. 3, pp. 297–305, Mar. 1991. doi: https://doi.org/10.1109/31.101322
Junguk Cho, Shahnam Mirzaei, Jason Oberg, and Ryan Kastner. 2009. Fpga-based face detection system using Haar classifiers. In Proceedings of the ACM/SIGDA international symposium on Field programmable gate arrays (FPGA '09). ACM, New York, NY, USA, 103–112. doi: http://dx.doi.org.prx.library.gatech.edu/10.1145/1508128.1508144.
Camera and Imaging Products Association, Design rule for Camera File system” DCF Version 2.0 (Edition 2010), Standardization Committee, Camera and Imaging Products Association, Apr 26, 2010.
S. Decker, D. McGrath, K. Brehmer and C. G. Sodini, “A 256×256 CMOS imaging array with wide dynamic range pixels and column-parallel digital output,” in IEEE Journal of Solid-State Circuits, vol. 33, no. 12, pp. 2081–2091, Dec 1998. doi: https://doi.org/10.1109/4.735551
A. Dickinson, B. Ackland, E. S. Eid, D. Inglis and E. R. Fossum, “Standard CMOS active pixel image sensors for multimedia applications,” in Proceedings Sixteenth Conference on Advanced Research in VLSI, Chapel Hill, NC, 1995, pp. 214–224. doi: https://doi.org/10.1109/ARVLSI.1995.515622
S. Dutta and W. Wolf, “A flexible parallel architecture adapted to block-matching motion-estimation algorithms,” in IEEE Transactions on Circuits and Systems for Video Technology, vol. 6, no. 1, pp. 74–86, Feb 1996. doi: https://doi.org/10.1109/76.486422
O. Elkhalili, O. M. Schrey, P. Mengel, M. Petermann, W. Brockherde and B. J. Hosticka, “A 4×64 pixel CMOS image sensor for 3-D measurement applications,” in IEEE Journal of Solid-State Circuits, vol. 39, no. 7, pp. 1208–1212, July 2004. doi: https://doi.org/10.1109/JSSC.2004.829927
R. D. Gow, David Renshaw, Keith Findlater, Stuart J. McLeod, John Hart, and Robert L. Hicol “A Comprehensive Tool for Modeling CMOS Image-Sensor-Noise Performance,” in IEEE Transactions on Electron Devices, vol. 54, no. 6, pp. 1321–1329, June 2007. doi: https://doi.org/10.1109/TED.2007.896718
Bahadir K. Gunturk, John Glotzbach, Yucel Altunbasak, Ronald W. Schafer, and Russel M. Mersereau, “Demosaicking: color filter array interpolation,” in IEEE Signal Processing Magazine, Jan 2005, pp. 44–54.
J. I. Guo, C. M. Liu and C. W. Jen, “The efficient memory-based VLSI array designs for DFT and DCT,” in IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing, vol. 39, no. 10, pp. 723–733, Oct 1992. doi: https://doi.org/10.1109/82.199898
Eric Hamilton, JPEG File Interchange Format, Version 1.02, Sept 1, 1992.
H. Hirschmuller, “Stereo Processing by Semiglobal Matching and Mutual Information,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 328–341, Feb 2008. doi: https://doi.org/10.1109/TPAMI.2007.1166
Technical Standardization Committee on AV & IT Storage Systems and Equipment, Exchangeable image file format for digital still cameras: Exif Version 2.2, Japan Electronics and Information Technology Industries Association, JEITA CP-3451, Apr, 2002.
S. Kappagantula and K.R. Rao, “Motion Compensated Predictive Interframe Coding,” in IEEE Transactions on Communications, vol. 33, no. 9, pp. 1011–1015, Sept 1985.
S. Kavadias, B. Dierickx, D. Scheffer, A. Alaerts, D. Uwaerts and J. Bogaerts, “A logarithmic response CMOS image sensor with on-chip calibration,” in IEEE Journal of Solid-State Circuits, vol. 35, no. 8, pp. 1146–1152, Aug 2000. doi: https://doi.org/10.1109/4.859503
Sujung Kim, Wook-joong Kim and Seong-Dae Kim, “Automatic white balance based on adaptive feature selection with standard illuminants,” in 2008 15th IEEE International Conference on Image Processing, San Diego, CA, 2008, pp. 485–488. doi: https://doi.org/10.1109/ICIP.2008.4711797
S. Kleinfelder, SukHwan Lim, Xinqiao Liu and A. El Gamal, “A 10000 frames/s CMOS digital pixel sensor,” in IEEE Journal of Solid-State Circuits, vol. 36, no. 12, pp. 2049–2059, Dec 2001. doi: https://doi.org/10.1109/4.972156
Kodak, Basic Photographic Sensitometry Workbook, document H-740, Eastman Kodak Company, November 2006.
T. Koizumi, Hwan-Sul Chun and H. Zen, “A new optical detector for a high-speed AF control,” in IEEE Transactions on Consumer Electronics, vol. 42, no. 4, pp. 1055–1061, Nov 1996. doi: https://doi.org/10.1109/30.555905
C. W. Kok, “Fast algorithm for computing discrete cosine transform,” in IEEE Transactions on Signal Processing, vol. 45, no. 3, pp. 757–760, Mar 1997. doi: https://doi.org/10.1109/78.558495
T. Komarek and P. Pirsch, “Array architectures for block matching algorithms,” in IEEE Transactions on Circuits and Systems, vol. 36, no. 10, pp. 1301–1308, Oct 1989. doi: https://doi.org/10.1109/31.44346
Toshiharu Kondo, Akihiro Kikuchi, Takashi Kohashi, Fumiaki Kato, and Katuaki HIrota, “Digital color video camera with auto-focus, auto-exposure, and auto-white balance, and an auto exposure system therefor which compensates for abnormal lighting,” U. S. Patent 5,093,716, Mar 3, 1992.
Kosov S., Thormählen T., Seidel HP. (2009) Accurate Real-Time Disparity Estimation with Variational Methods. In: Bebis G. et al. (eds) Advances in Visual Computing. ISVC 2009. Lecture Notes in Computer Science, vol. 5875. Springer, Berlin, Heidelberg
Edwin H. Land, “Absolute one-step photography,” in Photographic Science and Engineering, vol. 16, no. 4, July–Aug 1972, pp. 247–257.
Yung-Pin Lee, Thou-Ho Chen, Liang-Gee Chen, Mei-Juan Chen and Chung-Wei Ku, “A cost-effective architecture for 8×8 two-dimensional DCT/IDCT using direct method,” in IEEE Transactions on Circuits and Systems for Video Technology, vol. 7, no. 3, pp. 459–467, Jun 1997. doi: https://doi.org/10.1109/76.585925
R. Lienhart and J. Maydt, “An extended set of Haar-like features for rapid object detection,” in Proceedings. International Conference on Image Processing, 2002, pp. I-900-I-903 vol. 1. doi: https://doi.org/10.1109/ICIP.2002.1038171
Reoxiang Li, Bing Zeng and M. L. Liou, “A new three-step search algorithm for block motion estimation,” in IEEE Transactions on Circuits and Systems for Video Technology, vol. 4, no. 4, pp. 438–442, Aug 1994. doi: https://doi.org/10.1109/76.313138
Douglas O. Linder, “Famous Trials: Oklahoma City Bombing Trial,” http://law2.umkc.edu/faculty/projects/ftrials/mcveigh/mcveightrial.html, 2006, accessed Dec 20, 2016.
B. Liu and A. Zaccarin, “New Fast Algorithms for the Estimation of Block Motion Vectors,” in IEEE Transactions on Circuits and Systems for Video Technology, vol. 3, no. 2, pp. 148–157, Apr. 1993.
C. Loeffler, A. Ligtenberg and G. S. Moschytz, “Practical fast 1-D DCT algorithms with 11 multiplications,” in International Conference on Acoustics, Speech, and Signal Processing,, Glasgow, 1989, pp. 988-991 vol. 2. doi: https://doi.org/10.1109/ICASSP.1989.266596
Richard F. Lyon and Paul M. Hubel, “Eyeing the Camera: Into the Next Century,” in Proc. IS&T/SID 10th Color Imaging Conf., 2002, pp. 349–355.
H. S. Malvar, A. Hallapuro, M. Karczewicz and L. Kerofsky, “Low-complexity transform and quantization in H.264/AVC,” in IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 598–603, July 2003. doi: https://doi.org/10.1109/TCSVT.2003.814964
Microsoft, Microsoft Extensible Firmware Initiative FAT32 File System Specification, Version 1.03, Dec 6, 2000.
Junichi Nakamura, ed., Image Sensors and Signal Processing for Digital Still Cameras, CRC Press, 2005.
A. N. Netravali and J. D. Robbins, “Motion-compensated television coding: Part I,” in The Bell System Technical Journal, vol. 58, no. 3, pp. 631–670, Mar 1979. doi: https://doi.org/10.1002/j.1538-7305.1979.tb02238.x
R. H. Nixon, S. E. Kemeny, B. Pain, C. O. Staller and E. R. Fossum, “256×256 CMOS active pixel sensor camera-on-a-chip,” in IEEE Journal of Solid-State Circuits, vol. 31, no. 12, pp. 2046–2050, Dec 1996. doi: https://doi.org/10.1109/4.545830
NVIDIA, NVIDIA GeForce GTX 980, white paper, 2014.
NVIDIA, NVIDIA Jetson TX1 System-on-Module, data sheet, 2014.
Kouji Oizumi, Nozomu Kitagishi, and Shoichi Yamzaki, “Optical system for stabilizing an image,” U. S. Patent 5,270,857, Dec 14, 1993.
Kenneth A. Parulski and James E. McGarvey, “Automatic camera exposure control using variable exposure index CCD sensor,” U. S. Patent 5,610,654, March 11, 1997.
Lai-Man Po and Wing-Chung Ma, “A novel four-step search algorithm for fast block motion estimation,” in IEEE Transactions on Circuits and Systems for Video Technology, vol. 6, no. 3, pp. 313–317, Jun 1996.
R. K. Reich et al., “Integrated electronic shutter for back-illuminated charge-coupled devices,” in IEEE Transactions on Electron Devices, vol. 40, no. 7, pp. 1231–1237, Jul 1993. doi: https://doi.org/10.1109/16.216426
Carlo H. Se´quin and Michael F. Tompsett, Charge Transfer Devices, New York: Academic Press, 1975.
M. Schanz, C. Nitta, A. Bussmann, B. J. Hosticka and R. K. Wertheimer, “A high-dynamic-range CMOS image sensor for automotive applications,” in IEEE Journal of Solid-State Circuits, vol. 35, no. 7, pp. 932–938, July 2000. doi: https://doi.org/10.1109/4.848200
Norman L. Stauffer, “Auto-focus camera with solid state range finder,” U. S. Patent 3.945,023, Mar 16, 1976.
D. Stoppa, A. Simoni, L. Gonzo, M. Gottardi and G. F. Dalla Betta, “Novel CMOS image sensor with a 132-dB dynamic range,” in IEEE Journal of Solid-State Circuits, vol. 37, no. 12, pp. 1846–1852, Dec 2002. doi: https://doi.org/10.1109/JSSC.2002.804347
G. J. Sullivan and R. L. Baker, “Rate-distortion optimized motion compensation for video compression using fixed or variable size blocks,” Global Telecommunications Conference, 1991. GLOBECOM '91. 'Countdown to the New Millennium. Featuring a Mini-Theme on: Personal Communications Services, Phoenix, AZ, 1991, pp. 85–90 vol.1. doi: https://doi.org/10.1109/GLOCOM.1991.188361
S. M. Sze, Physics of Semiconductor Devices, second edition, New York: John Wiley and Sons, 1981.
Hiroaki Takada, editor, and Ken Sakamura, supervisor, ITRON4.0 Specification, version 4.00.00, Tron Association, 2002.
Texas Instruments, AM572x Sitara TM Processors Silicon Revision 2.0, 1.1 Technical Reference Manual, Literature Number SPRUHZ6H, Oct 2014, revised Nov 2016.
Jo Yew Tham, S. Ranganath, M. Ranganath and A. A. Kassim, “A novel unrestricted center-biased diamond search algorithm for block motion estimation,” in IEEE Transactions on Circuits and Systems for Video Technology, vol. 8, no. 4, pp. 369–377, Aug 1998. doi: https://doi.org/10.1109/76.709403
T. Theocharides, G. Link, N. Vijaykrishnan, M. J. Irwin and W. Wolf,. “Embedded hardware face detection,” in 17th International Conference on VLSI Design. Proceedings., 2004, pp. 133–138. doi: https://doi.org/10.1109/ICVD.2004.1260915
A. Theuwissen, “CMOS image sensors: State-of-the-art and future perspectives,” in ESSCIRC 2007 – 33rd European Solid-State Circuits Conference, Munich, 2007, pp. 21–27. doi: https://doi.org/10.1109/ESSCIRC.2007.4430242
Hui Tian, Boyd A. Fowler, Abbas El Gamal; Analysis of temporal noise in CMOS APS. In Proc. SPIE 3649, Sensors, Cameras, and Systems for Scientific/Industrial Applications, 177 (Apr 27, 1999); doi:https://doi.org/10.1117/12.347073.
Hui Tian, Abbas El Gamal, Analysis of 1/f noise in CMOS APS. In Proc. SPIE 3965, Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications, 168 (May 15, 2000); doi:https://doi.org/10.1117/12.385433.
Albert H. Titus, Maurice C.-K. Cheung, and Vamsy P. Chodavarapu, “CMOS Photodetectors,” Chapter 4 in Jeong-Woo Park, ed., Photodiodes/Book 2, July 2011.
F. Tombari, S. Mattoccia, L. Di Stefano and E. Addimanda, “Classification and evaluation of cost aggregation methods for stereo correspondence,” in 2008 I.E. Conference on Computer Vision and Pattern Recognition, Anchorage, AK, 2008, pp. 1–8. doi: https://doi.org/10.1109/CVPR.2008.4587677
Akio Tsuji and Teruo Sano, “Electronic camera with automatic exposure control,” U. S. Patent 5,223,935, June 29, 1993.
Christos Ttofis, Christos Kyrkou, and Theocharis Theocharides. 2015. “A Hardware-Efficient Architecture for Accurate Real-Time Disparity Map Estimation,” in ACM Transactions on Embedded Computing Systems, vol. 14, no. 2, Article 36 (Feb 2015), pp. 26. doi: http://dx.doi.org.prx.library.gatech.edu/10.1145/2629699
P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in Proceedings of the 2001 I.E. Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, 2001, pp. I-511-I-518 vol.1. doi: https://doi.org/10.1109/CVPR.2001.990517
Gregory K. Wallace. 1991. “The JPEG still picture compression standard,” in Communications of the ACM, vol. 34, no. 4 (Apr 1991), pp. 30–44. doi: http://dx.doi.org.prx.library.gatech.edu/10.1145/103085.103089
Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” in IEEE Transactions on Image Processing, vol. 13, no. 4, Apr 2004, pp. 600–612.
M. H. White, D. R. Lampe, F. C. Blaha and I. A. Mack, “Characterization of surface channel CCD image arrays at low light levels,” in IEEE Journal of Solid-State Circuits, vol. 9, no. 1, pp. 1–12, Feb 1974. doi: https://doi.org/10.1109/JSSC.1974.1050448
Nathan Whitehead and Alex Fit-Florea, “Precision & performance: floating point and IEEE 754 compliance for NVIDIA GPUs,” NVIDIA, 2011.
T. Wiegand, G. J. Sullivan, G. Bjontegaard and A. Luthra, “Overview of the H.264/AVC video coding standard,” in IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 560–576, July 2003. doi: https://doi.org/10.1109/TCSVT.2003.815165
Wayne Wolf, Ahmed A. Jerraya, and Grant Martin, “Multiprocessor System-on-Chip (MPSoC) Technology,” in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 27, no. 10, Oct 2008, pp. 1701–1713.
Marilyn Wolf, The Physics of Computing, Cambridge MA: Elsevier, 2017.
R. Xu, B. Liu and J. Yuan, “A 1500 fps Highly Sensitive 256 X 256 CMOS Imaging Sensor With In-Pixel Calibration,” in IEEE Journal of Solid-State Circuits, vol. 47, no. 6, pp. 1408–1418, June 2012. doi: https://doi.org/10.1109/JSSC.2012.2192662
Orly Yadid-Pecht, Karmak Mansoorian, Eric R. Fossum, Bedabrata Pain; Optimization of noise and responsivity in CMOS active pixel sensors for detection of ultralow-light levels. Proc. SPIE 3019, Solid State Sensor Arrays: Development and Applications, 125 (Apr 25, 1997); doi: https://doi.org/10.1117/12.275185.
Akira Yamanaka and Toshinori Imura, “Auto-focus camera having a rangefinder,” U. S. Patent 4,300,823, Nov 17, 1981.
K. M. Yang, M. T. Sun and L. Wu, “A family of VLSI designs for the motion compensation block-matching algorithm,” in IEEE Transactions on Circuits and Systems, vol. 36, no. 10, pp. 1317–1325, Oct 1989. doi: https://doi.org/10.1109/31.44348
Ming-Hsuan Yang, D. J. Kriegman and N. Ahuja, “Detecting faces in images: a survey,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 1, pp. 34–58, Jan 2002. doi: https://doi.org/10.1109/34.982883
Shengqi Yang, Wayne Wolf and Narayanan Vijaykrishnan, “Power and performance analysis of motion estimation based on hardware and software realizations,” in IEEE Transactions on Computers, vol. 54, no. 6, June 2005, pp. 714–726.
Ramin Zabih and John Woodfill, “Non-parametric local transforms for computing visual correspondence,” in Jan-Olof Eklundh, ed., Computer Vision --- ECCV '94: Third European Conference on Computer Vision Stockholm, Sweden, May 2–6 1994 Proceedings, Volume II, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 151–158, doi https://doi.org/10.1007/BFb0028345
R. Zheng, T. Wei, D. Gao, Y. Zheng, F. Li and H. Zeng, “Temporal noise analysis and optimizing techniques for 4-T pinned photodiode active pixel sensor,” in 2011 I.E. International Conference on Signal Processing, Communications and Computing (ICSPCC), Xi'an, 2011, pp. 1–5. doi: https://doi.org/10.1109/ICSPCC.2011.6061804
Sang-Min Lee, Hyunsik Park, and Bruce A. Wooley, “Per-pixel floating-point ADCs with electronic shutters for a high dynamic range, high frame rate infrared focal plane array,” in IEEE 2006 Custom Integrated Circuits Conference, IEEE, 2006, pp. 647-650.
A. I. Krymski, N. E. Bock, N. Tu, D. Van Blerkom, and E. R. Fossum, “A high-speed, 240-frame/s, 4.1-Mpixel CMOS sensor,” IEEE Transactions on Electron Devices, 50(1), January 2003, pp. 130-135.
Rene Helbing, Still image stabilization suitable for compact camera environments,” U. S. Patent Application Publication US 2008/0030587, February 7, 2008.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this chapter
Cite this chapter
Wolf, M. (2018). Image Capture Systems and Algorithms. In: Smart Camera Design. Springer, Cham. https://doi.org/10.1007/978-3-319-69523-5_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-69523-5_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-69522-8
Online ISBN: 978-3-319-69523-5
eBook Packages: EngineeringEngineering (R0)