1 Introduction

Imaging systems are important tools in various scientific and engineering fields. In the conventional imaging approach, an object is directly imaged on an image sensor, as shown in Fig. 1a. The imaging optics is designed to make the captured image as identical as possible to the object, and the size and cost of the optics are increased to compensate for aberrations. On the other hand, computational imaging is a novel framework for constructing imaging systems by orchestrating optics and information science, as shown in Fig. 1b [1]. The optics is designed for encoding the object as the captured image. The object is reconstructed from the captured image with a decoding computational process. Based on this framework, we can minimize the optical hardware, enhance the imaging performance of optical systems, and visualize invisible targets.

Wavefront sensing and control techniques, such as digital holography and computer-generated holography, serve as the foundational principles of computational imaging [2,3,4]. In addition, Fourier transform profilometry and multi-aperture imaging represent other origins of this field [5,6,7]. These renowned computational imaging techniques have contributed to advances in various fields, including life-science, industrial engineering, and machine vision. Besides recent advances in optical technologies, image sensors, and computers, state-of-the-art technologies in information science, such as compressive sensing and machine learning, have also stimulated the field of computational imaging, where randomness takes an important role due to its redundancy and orthogonality [8,9,10,11].

Fig. 1
figure 1

Schematic diagrams of a conventional imaging and b computational imaging

Full size image

2 Quantitative phase imaging

Quantitative phase imaging is important for visualizing transparent biological specimens in biomedicine and life science, without invasive fluorescent staining [12, 13]. Digital holography is an established technique for quantitative phase imaging and is one of the longstanding research topics in the field of optical imaging [2, 4]. One drawback with digital holography is the requirement for interferometric measurements, which makes the optical setup bulky and complicated.

Another established technique for quantitative phase imaging is diffraction imaging [14, 15]. In diffraction imaging, the complex amplitude field of an object is reconstructed from a single diffraction intensity image by a phase retrieval algorithm [16,17,18]. As a result, interferometric measurements are not necessary, and compact optical hardware can be implemented. However, a limited size of the object, which is called the support, must be assumed in phase retrieval to solve the ill-conditioned inverse problem. Ptychography is a diffraction imaging technique for extending the field-of-view by performing multiple measurements through a scanning process; however, it is not applicable to dynamic scenes [19,20,21].

To solve the above-mentioned issues in single-shot and multi-shot diffraction imaging, we have presented single-shot, support-free diffraction imaging with coded modulation, as shown in Fig. 2a [22,23,24]. In this method, a complex amplitude object is illuminated with coherent light, and light from the object is captured through a coded aperture, as shown in Fig. 2b [25]. The coded aperture is composed of randomly aligned pinholes to improve the condition of the inverse problem. Our method measures a single intensity image, as shown in Fig. 2c, without interferometric measurements, and therefore, its optical setup is simple and compact compared with those of digital holography. Instead of the coded aperture, randomly structured illumination can also be applied to our single-shot diffraction imaging method [24]. Both the amplitude and phase of the object are reconstructed from the single intensity image by using a phase retrieval algorithm based on compressive sensing, as shown in Fig. 2d, e, respectively [26,27,28]. The pixel count was \(740^2\) and the field of view was \(3.4~\textrm{mm} \times 3.4~\textrm{mm}\). We extend our single-shot diffraction imaging with coded modulation to single-shot multi-dimensional imaging and single-pixel imaging [29,30,31,32,33].

Fig. 2
figure 2

Single-shot diffraction imaging with coded modulation. a Optical setup with coded aperture. b Coded aperture. c Captured intensity image of a mixture of water and oil. Reconstructed d amplitude and e phase. Adapted with permission from [25] © Optica Publishing Group

Full size image

3 Computer-generated holography

Computer-generated holography (CGH) is a light control technique for computationally calculating an interference pattern—called a hologram—that reproduces an arbitrary optical field [34]. CGH is promising for laser processing in precision engineering, optical tweezers and stimulation in life science, and three-dimensional displays and near-eye displays for next-generation visual interfaces [35,36,37]. It is possible to dynamically control optical fields with computer-generated holography by means of a spatial light modulator [38]. However, commercially available spatial light modulators control either the amplitude or the phase of light waves. Therefore, iterative algorithms, such as the Gerchberg–Saxton method, have been used to synthesize amplitude-only or phase-only holograms [39]. The iterative process in the algorithms has prevented applications of computer-generated holography in real-time and interactive situations.

To solve this issue, we introduced deep learning to computer-generated holography for non-iterative hologram synthesis [18, 40, 41]. One of the schemes for computer-generated holography based on deep learning is shown in Fig. 3. A convolutional deep neural network is trained to realize the inverse process of the optical propagation. The training dataset consists of pairs of random input patterns and their computationally propagated results, which are speckle output patterns. Then, the network can synthesize a hologram that optically reproduces an arbitrary target pattern. As a result, the computational speed for synthesizing holograms is two orders of magnitude faster using deep learning, compared to a conventional iterative approach [41]. Nowadays, deep learning approaches are recognized as foundational techniques in computer-generated holography [42,43,44,45].

Fig. 3
figure 3

Computer-generated holography based on deep learning. Adapted with permission from [40] © Optica Publishing Group

Full size image

Another challenge in computer-generated holography is the requirement for highly coherent light. This often necessitates the use of large and expensive light sources, which also can potentially be invasive to the eyes. We address this problem by introducing spatiotemporally incoherent light to computer-generated holography [46]. An issue of computer-generated holography with incoherent light is the high computational cost due to many modes in the incoherent light propagation. In our method, incoherent light is described as a set of coherent random wavefronts based on stochastic gradient descent [47]. The experimental demonstration of our method by using a chip-on-board white light-emitting diode with a diameter of 2.3 cm is shown in Fig. 4. The color image in Fig. 4a is optically reproduced as in Fig. 4b with the two-layered monochrome hologram cascade composed of the first amplitude hologram in Fig. 4c  and the second phase hologram in Fg. 4d. The size of the reproduced image was \(2.8~\textrm{mm} \times 2.8~\textrm{mm}\). This approach has been extended to diffractive optics design [48].

Fig. 4
figure 4

Incoherent computer-generated holography. a Target color image and b its reproduced result. c First amplitude hologram and d second phase hologram in two-layered monochrome hologram cascade. Adapted with permission from [46] © Optica Publishing Group

Full size image

4 Imaging through scattering media

Imaging through scattering media is a longstanding issue in the field of optics for various applications, including biomedicine, astronomy, and security. Recent advances in optics and information science have enabled visualization inside or behind strongly scattering media, where ballistic photons are very few [49,50,51,52,53]. Although various methods have been established for imaging through scattering media, speckle-correlation imaging has advantages over other approaches in terms of its noninvasiveness, single-shot capability, and simple optical hardware [54, 55]. Speckle-correlation imaging utilizes a shift-invariance of the scattering random impulse response, which is called the memory effect [56, 57].

We extended speckle-correlation imaging to multidimensional cases. Depth imaging and spectral imaging through scattering media were realized by utilizing the scale-invariance of the scattering response along the axial and spectral dimensions, respectively [58,59,60]. The scattering scale-invariances in the first and second cases are called the axial memory effect and spectral (chromatic) memory effect, respectively [61, 62]. The experimental results from spectral speckle-correlation imaging are shown in Fig. 5. Point sources with a diameter of 0.1 mm and wavelengths of 520 nm and 540 nm shown in Fig. 5a are captured through scattering media, as shown in Fig. 5b. The two-color point sources are recovered and spectrally resolved as shown in Fig. 5c.

Fig. 5
figure 5

Spectral speckle-correlation imaging. a Multicolor object. b Capture image through scattering media. c Reconstruction image. Adapted with permission from [60] © Optica Publishing Group

Full size image

An issue with speckle-correlation imaging is the limited field-of-view because the range of the memory effect becomes small in a thick scattering medium [54, 55, 63]. To address this issue, we incorporated extrapolation of the speckle correlation into the reconstruction process of speckle-correlation imaging [64]. An experimental demonstration of extrapolated speckle-correlation imaging with an untrained deep neural network called deep image prior is shown in Fig. 6 [65, 66]. The object and its image captured through scattering media without imaging optics are shown in Fig. 6a, b, respectively. The conventional approach without the extrapolation cannot recover the whole object, as shown in Fig. 6c. On the other hand, the field-of-view of the reconstruction result obtained by the proposed approach is extended compared with the conventional case, as shown in Fig. 6d, where the pixel count was \(60^2\).

Fig. 6
figure 6

Extrapolated speckle-correlation imaging with deep image prior. a Object. b Image captured through scattering media. Reconstruction results c without and d with the extrapolation of speckle correlation. Adapted with permission from [65] © Optica Publishing Group

Full size image

Blind deconvolution is a noninvasive technique for imaging through scattering media [67, 68]. Blind deconvolution is applicable to scattering media with limited-sized impulse responses but is not applicable to scattering media with random impulse responses. On the other hand, speckle-correlation imaging mentioned above is applicable to scattering media with random impulse responses but is not applicable to scattering media with limited-size impulse responses. One issue with single-shot blind deconvolution is instability of the reconstruction process because it needs to solve an ill-posed inversion problem to estimate both the object and the impulse response from a single captured image.

To solve this issue, we introduce a coded aperture to single-shot blind deconvolution, where the coded aperture reduces unknown variables on the aberrated pupil plane and improves the stability of the reconstruction process [69]. The experimental demonstration is shown in Fig. 7. The object in Fig. 7a is composed of point sources, and it is captured under a severe defocus condition with incoherent light, as shown in Fig. 7b. In this case, the blind deconvolution algorithm does not work, as shown in Fig. 7c. The image captured through the coded aperture is shown in Fig. 7d. In the case with the coded aperture, the object is reconstructed well, as shown in Fig. 7e, where the pixel count was \(200^2\). We have extended this approach to quantitative phase imaging through scattering media by introducing coherent light [70].

Fig. 7
figure 7

Single-shot blind deconvolution with coded aperture. a Object. b Defocused image without the coded aperture and c its reconstruction result. d Defocused image with the coded aperture and e its reconstruction result. Adapted with permission from [69] © Optica Publishing Group

Full size image

Digital optical phase conjugation, which is also known as time reversal, enables light shaping behind a scattering medium, [71,72,73]. In digital optical phase conjugation, an object inside a scattering medium is illuminated with coherent light, and light from scattering medium is captured with a wavefront sensor. The scattering medium is holographically illuminated from the outside by the conjugation of the captured wavefront, and then the object inside the scattering medium is optically reproduced. A coherent light source, wavefront sensor, and wavefront reproducer, which contain an interferometric setup and a spatial phase modulator, are necessary to realize digital optical phase conjugation.

We developed digital optical phase conjugation with incoherent light and without the wavefront sensor or reproducer [74]. The experimental demonstration is shown in Fig. 8. The object in Fig. 8a is illuminated with spatiotemporally incoherent light. The image captured through a scattering medium without imaging optics is shown in Fig. 8b. The captured image is displayed on a display device with incoherent light in front of the scattering medium. The light intensity behind the scattering medium is shown in Fig. 8c. The contrast of the reproduced image is low because of the non-negativity and the realness of the incoherent light. To address this, we introduce background suppression by random pixel shuffling of the captured image in Fig. 8b to optically reproduce the background. The result with the background suppression is shown in Fig. 8d, where the contrast is significantly improved.

Fig. 8
figure 8

Incoherent optical phase conjugation. a Object. b Image captured through scattering media. Optically reproduced results c without and d with background suppression. Adapted with permission from [74] © Optica Publishing Group

Full size image

5 Conclusion

We presented several computational imaging techniques with random optical modulation, including quantitative phase imaging, computer-generated holography, and imaging through scattering media. Information science has played a pivotal role in the progress of computational imaging, particularly where randomness is a significant factor in improving imaging performance. On the other hand, recent advances in optical and sensor devices, such as metalenses and event cameras, open the possibility of offering novel degrees of freedom for designing computational imaging systems [75,76,77]. Insights that span over these fields have the potential to further innovate computational imaging.