Multi-aperture optics as a universal platform for computational imaging
Computational imaging is a novel imaging framework based on optical encoding and computational decoding. To avoid a heuristic design that depends on the particular problem to be solved, multi-aperture optics is useful as a universal platform for optical encoding. In this paper, the fundamental properties of multi-aperture optics are summarized. Then some examples of interesting functions implemented by multi-aperture optics are explained, together with some effective applications.
KeywordsMulti-aperture optics Compound-eye imaging Computational imaging Image processing Compressive sensing
Progress made in information technology has enabled the realization of a novel imaging framework called computational photography or computational imaging . Dynamic range extension  and depth-of-field extension  are practical examples of the benefits achieved using this framework. A plenoptic camera  is also an attractive example demonstrating unique functionalities, such as refocusing and the ability to translate the viewing point after image capturing.
The fundamental procedure of computational imaging consists of optical encoding of the object signals and computational decoding for image reconstruction. Combinatorial variations of these processes provide a variety of imaging modalities. Optical encoding is achieved by various methods, such as specific optical devices , mechanical scanning , pattern illumination  and scattering media . Once the optical signals are encoded, computational decoding is performed by digital processing on a computer. Although the resources and the processing throughput of the digital processing should be taken into account, optical encoding is a key process in computational imaging for practical applications.
Most optical encoding techniques are designed by a heuristic approach suitable for a given problem. However, such a design process makes it difficult to achieve a systematic design that is applicable to a wide range of problems. To alleviate this situation, a versatile optical platform capable of flexible optical encoding is desired. Multi-aperture optics is a promising and effective solution to this problem.
In this paper, the fundamental properties of multi-aperture optics are summarized to illustrate this technique’s suitability as a universal platform for computational imaging. Some examples of interesting functions implemented by multi-aperture optics are explained. Then, applications of the multi-aperture optical systems are demonstrated to show their effectiveness and extendibility.
2 Computational imaging
Dowski and Cathy proposed an interesting scheme to extend the depth of field of imaging optics . In their method, the point spread function (PSF) of the imaging system is intentionally modified so as to be insensitive to defocus. A phase plate that induces a cubic phase retardation is introduced for this purpose. Because the decoder knows the modified PSF, the blurred image is restored by deconvolution with the PSF. This method achieves not only image deblurring, but also extended depth of field.
Demonstration examples of computational imaging
Dynamic range extension
Light field capturing
Light field capturing
CDI, CS decoder
3 Multi-aperture optics
With respect to the fields of view of the individual elementary optics, two types of designs are possible: field division and field overlapping. The former divides the field of view using the individual elementary optics, whereas in the latter, all elementary optics observe almost the same field of view. Field division is suitable for imaging applications because simple stitch processing is enough to reconstruct the observed field. It is also effective for wide field-of-view observation. On the other hand, field overlapping requires elaborate processing to retrieve the object information, but different properties of signals can be captured at a time. It is useful for multi-dimensional signal observation.
3.2 Optical properties
A compound-eye imaging system known as TOMBO , shown in Fig. 4, is a typical example of the divided type of multi-aperture optics. Division of the imaging optics provides interesting optical properties. For \(N^2\)-divided optics under the condition of equivalent F-number of the imaging lens and equivalent image sensor, the focal length is reduced by 1/N. The image sensor is divided into \(N^2\) sections, so that the pixel number of a unit image reduced by \(1/N^2\). This enlarges the pixel size relative to the object, and the permissible circle of confusion increases. As a result, the depth of field is increased N-times.
Introducing an array of pinholes or microlenses into the imaging optics is an easy way to implement multi-aperture optics. A plenoptic camera  is a typical instance. A simpler implementation is just to set an array of microlenses in front of the image sensor to focus on the sensor surface. A thin wafer-level camera  and TOMBO  are typical examples. As a variant of multi-aperture optics, a sophisticated phase imaging scheme in which an aperture array is inserted into the light propagation field has been proposed .
4 Functions achieved by multi-aperture optics
Examples of functions achieved by multi-aperture optics
Light field capturing
Elimination of main lens
Extended depth of field
4.1 Geometrical measurement
Because multiple elementary optics observe the object from different positions, parallax signals can be obtained for geometrical measurements. Multiple baselines with different lengths and orientations for stereo matching can be assigned to the observations, which improves the measurement performance . Note that, in the case of multi-aperture optics in a regular arrangement and having the same focal length, the images observed by all elementary optics become identical at infinity. This considerably reduces the total amount of observation signals. To alleviate this situation, an irregular arrangement of the elementary optics is effective . It was demonstrated that the imaging properties for observation of a distant object were improved with a slightly disordered arrangement of elementary imaging optics.
4.2 Combinatorial measurement
Using a heterogeneous configuration, various optical properties can be assigned to the individual elementary optics. For example, optical filters with different intensity attenuations, wavelength transmittances, and polarization orientations are utilized to capture the corresponding optical properties. By combining the different observations, various kinds of object information can be retrieved. Spectral imaging is achieved by placing wavelength filters on the individual elementary optics . Gonio-imaging just utilizes the geometrical displacement of the apertures . With an appropriate decoding processing, such as using a compressive sensing decoder , it is possible to retrieve not only exclusive separable signals but also inclusive mixed ones.
4.3 Focus sweeping
Compact integration of multiple functions is an attractive feature of multi-aperture optics for product development. In this section, some examples of sophisticated applications based on multi-aperture optical systems are presented.
5.1 Intra-oral diagnosis system
5.2 Single-shot phase imaging
An aperture array inside a light propagation field works as a sieve that samples the observation field to reduce the complexity of the decoding process and to extend the field of view. This technique has been proposed for single-shot phase imaging with a coded aperture, called SPICA. SPICA employs compressive Fresnel holography  and coherent diffraction imaging  iteratively to retrieve the amplitude and phase signals of the object. This technique is based on a variant of multi-aperture optics and shows the potential of this optical system.
5.3 Wide-field and deep-focused imaging
A superposition eye was applied to wide-field and deep-focused imaging . In addition to the focus sweep property, point symmetry of the optics enables omnidirectional observation, as shown in Fig. 5. Image blur caused by a shift-invariant PSF can be restored with deconvolution, which extends the depth of field and widens the field of view.
5.4 Optical system virtualization
Multi-aperture optics is promising as a universal platform for computational imaging because of its versatility in optical signal encoding. Division of baseline optics is an effective strategy for realizing multi-aperture optics owing to the present capabilities of hardware miniaturization and functional integration. Suitable design and fabrication methodologies for multi-aperture optics will be required to promote widespread use of this promising technology.
- 4.Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., Hanrahan, P.: Light field photography with a hand-held plenoptic camera. Comput. Sci. Tech. Rep. CSTR 2(11), 1–11 (2005)Google Scholar
- 14.Plemmons, R.J., Prasad, S., Matthews, S., Mirotznik, M., Barnard, R., Gray, B., Pauca, V.P., Torgersen, T.C., van der Gracht, J., Behrmann, G.: PERIODIC: integrated computational array imaging technology. In: Adaptive optics: analysis and methods/computational optical sensing and imaging/information photonics/signal recovery and synthesis topical meetings on CD-ROM, OSA Technical Digest (CD) (Optical Society of America), paper CMA1 (2007)Google Scholar
- 22.Tsumura, N., Haneishi, H., Miyake, Y.: Estimation of spectral reflectances from multi-band images by multiple regression analysis. Jpn. J. Opt. 27, 384 (1998). (in Japanese) Google Scholar