Introduction

The spatial image reproduction three-dimensional (3D) display methods can display natural 3D images. They can reconstruct spatial images in a similar manner to when objects are being directly observed. These methods can be achieved using integral photography1,2,3 or holography. Integral photography is a method that reproduces discrete light rays from objects. Electronic holography is considered to be the ideal and ultimate 3D video display method since it can reconstruct the wavefronts of the light reflected by objects and propagated through space. There exist advantages and disadvantages of each method. It is desirable for both the image size of a 3D image and the viewing-zone angle to be large. However, there exist many technical problems to overcome. One of the most important problems remaining to be solved for electronic holography is enlargement of the image size.

A wide range of research has been conducted concerning 3D displays using electronic holography for reproducing 3D videos. This includes research directed towards increasing the image size4,5,6 and research directed towards real-time operation using conventional central processing units (CPUs)7 or graphics processing units (GPUs)8,9,10 for hologram calculations.

Various methods for increasing the viewing zone by using space-division multiplexing have been proposed11,12,13. These methods are affected by the problem that part of the viewing-zone angle is missing. To overcome this problem, a method has been proposed that uses a beam splitter and multiple spatial light modulators (SLMs)14. However, the viewing-zone angle is scalable only in the horizontal direction and each of the gaps between SLMs must equal the width of an SLM.

On the other hand, various methods of increasing the image size by using space-division multiplexing have also been proposed. Some problems with these proposals are that they only provide parallax in the horizontal direction15,16, the image size is only scalable in the horizontal direction17, or high-refresh-rate SLMs are required because the display frame rate is inversely proportional to the space-division multiplexing count18.

In a previous paper, we proposed a method of increasing the image size using space-division multiplexing by locating an optical system containing a lens array and other components in front of multiple SLMs6. Various methods of colourizing reconstructed images by using space-division multiplexing have been proposed19,20,21,22. Although these methods achieve ideal colourization of the reconstructed 3D images without colour breakup, they are too complex to adapt to the multiple-SLM system mentioned above because they will require triple the number of SLMs than in a monochromatic system for static colourization and they will also require a large or complex combination of optical systems. We proposed a colourization technique23,24 based on time-division multiplexing25,26,27,28,29,30,31,32 using three lasers (i.e., red, green and blue) and a set of three revolving wheel shutters synchronized with 60 Hz driven SLMs.

However, a problem with the method mentioned above is the increased size of the image-readout optical system caused by increasing the number of SLMs to obtain a larger image. The enlargement of this assembly in the direction of the SLM's surface is acceptable, but the enlargement in the direction of its optical axis causes a problem. The proposed method can be scaled to increase the number of SLMs for further enlargement of the image, but there exist three factors limiting the image size. Firstly, the sizes of the lenses, secondly, the physical path length between SLMs and lenses to install the image-readout optical system and thirdly, vignetting of the first-order diffraction beam by the outer edge of the polarization beam splitter (PBS) that was located at a distance from the SLMs. A solution for the second and third problems is to reduce the number of PBS steps to shorten the path length in the image-readout optical system.

In this paper, we propose a method of producing larger colour electronic holographic images with a newly devised space-saving image-readout optical system for multiple reflection-type SLMs. This optical assembly is designed so that the physical path length of the image-readout part is half that of the previous method and larger number of SLMs can be tiled. We apply this method with 16 (4 × 4) 4K×2K-pixel SLMs to display larger full-parallax colour holographic 3D video than with the previous system23,24.

Results

Limitation of the number of PBS steps in the image-readout optical system

Figure 1 shows the configuration of the developed optical systems and their experimental setup and Table 1 shows a list of parameters. They are examples of the case when the horizontal number of SLMs Kx is four. Time-divided colour parallel light from a coherent light source as shown in Figs. 1(a) and 1(c) is incident perpendicularly on SLMs Cm through PBSs in Figs. 1(b) and 1(d). Note that the superscript of the letter represents the location within the array. Figures 1(a) and 1(b) show plane figures of the optical systems in the horizontal direction. Consequently, this is so-called in-line holography. Amplitude modulation (AM) type reflective SLMs Cm are used for displaying hologram data. This kind of SLM creates modulation from the polarization rotation of incident light that corresponds to the input signals. Therefore, the combination of the SLM and each corresponding polarization-selective device (such as a PBS) results in the AM output of the light. The hologram data displayed on Cm are synchronized with the wheel shutters and changed to accommodate the alternating wavelengths of the light sources (i.e., red, green, or blue) every 1/60 second. Since the pixels are arranged in a grid pattern on a thin hologram, the light reflected on Cm contains a primary beam (first-order diffraction beam), conjugate beam (minus first-order diffraction beam), carrier beam (zero-order beam) and high-order diffraction beams of each of them. Since the object beam is generated as a primary beam, the other beams are unnecessary light and must be eliminated.

Table 1 System parameters
Full size table
Figure 1
figure 1

Configuration of optical systems and their experimental setup.

Light source optical system (a) and (c). Image-readout optical system and a part of gap reduction optical system (b) and (d). Overall view of optical systems (e). H.S. took the photographs.

Full size image

In the previous system, an image-readout optical system as shown as Fig. 2(a) was used. PBSs are installed in a step-wise manner. We set the physical path length a and the geometrical path length a′, which was optically compensated by the refractive index of the PBS, between Cm and lenses as follows:

where nD is the relative refractive index of the PBS material for air. The sizes (width and depth) of the PBSs are ξD. f1 and f2 are the focal lengths of lenses and , respectively. Equation (1) indicates that a′ equals the sum of a and the optical effect from the difference of the refractive index caused by a PBS in the optical path. The limitation of the number of PBS steps lDa that is limited by a is necessary to satisfy the following relation:

The thickness of the polarizers and lenses and the difference of nD according to wavelength play only small or negligible roles.

Figure 2
figure 2

Image-readout optical systems.

AM type reflective SLMs without polarization filters are used for displaying hologram data. Combination of the SLM and each corresponding PBS makes AM output of the light. The PBS in the outer position D4 of conventional image-readout optical system produces the vignetting angled part of the readout light from the corresponding SLM C4 (a). Illumination light enters from multiple directions (e.g., enters from opposite sides) to reduce the number of PBS steps (b). However, this system requires a complex incident optical system. Newly devised image-readout optical system, which consists of pairs of PBSs, SLMs, HWPs and polarizers (c) and its operating principles (d). The PBS and SLM at odd-numbered positions are Do and Co and at even-numbered positions are De and Ce. The number of PBS steps can be halved with illumination light entering the PBSs from a single direction.

Full size image

On the other hand, when the number of SLMs increased, the PBSs in the outer positions produce the vignetting angled part of the readout light from the corresponding SLMs (i.e., light with an angle between the zero- and first-order diffraction beams). To avoid this vignetting by the outermost PBS, limitations of the number of PBS steps lDx and lDy, which depend on the vignetting limits of the horizontal and vertical directions, respectively, are necessary to satisfy the following relations:

where the height of the PBSs is ηD and the sizes (width and height) of the SLMs are denoted by and . The angles of the first-order diffraction beam θ1x and θ1y are given as follows:

where the horizontal and vertical pixel pitches are px and py and the wavelength of the reproduced light is denoted by λ. Therefore, the overall limitation of the number of PBS steps lD, which depends on either the effect of the physical path length a (derived from f1 and f2: cf. formulas (1) and (2)) or vignetting of the first-order diffraction beam by the outermost PBS (cf. formulas (3) or (4)), is given as follows:

where the notation indicates the floor function. lD is equal to the limitation of the horizontal number of PBSs by the optical system in Fig. 2(a).

When ηD equals ξD, the contribution to lD by the effect of horizontal direction lDx is more dominant than the effect of vertical direction lDy in equation (7) because is larger than and θ1x equals θ1y in most cases. Furthermore, lDa, which depends on a, has enough margin to install PBSs in a step-wise manner if we set the values of f1 and f2 appropriately. It is clear from formulas (1) and (2) that we can easy to enlarge lDa by elongation of f1. Therefore, it can be said that most effective parameter in equation (7) is lDx.

The relations between ξD and the limitation of the horizontal number of PBSs determined by lD are drawn as graphs in Fig. 3, where the above-mentioned parameters are selected as the values shown in Table 1. The dark gray shaded area shows feasible numbers of PBS steps. When PBSs with size ξD = ηD = 28 mm are used, the upper limit of the number of steps is three and it is clear that we cannot arrange four PBSs with four steps regardless of whether we would like to arrange the SLMs 4 × 4 in a plane.

Figure 3
figure 3

The relations between the size of PBS ξD and the limitation of the horizontal number of PBSs (f1 = 125 mm, f2 = 350 mm, nD = 1.52, ηD = ξD, , , px = py = 4.8 μm and λ = λB = 473 nm).

The dashed lines show the relations of ξD to lDa (green line), which is the limitation by the path length a, lDx (blue line), which is the limitation by the vignetting limits of the horizontal direction, lDy (red line), which is the limitation by the vignetting limits of the vertical direction and lD (upper bound of gray shaded area), which is the overall limitation, by the conventional optical system shown in Fig. 2(a). The solid lines show the relations of ξD to 2lDa, 2lDx, 2lDy and 2lD by the space-saving optical systems shown in Figs. 2(b)–2(d).

Full size image

To overcome the limitation of image size scalability, it is necessary to increase the gap T′ between SLMs and use a large PBS, or to divide the beam from the light source and make it incident from multiple directions (e.g., enter from opposite sides) to reduce the number of PBS steps as shown in Fig. 2(b). However, the first method mentioned in the previous sentence necessitates huge optical components in the following optical system. For example, if we use PBSs with size ξD = 35 mm so that the upper limit of the number of steps is as high as five, the physical sizes of lenses and L3 (i.e., ξ2, η2 and ϕ3) would have to be 1.25 times larger than the values shown in Table 1. Furthermore, the second method requires many optical components to create a complex incident light path. These methods are feasible but difficult or complicated.

Therefore, we devised an optical system in which illumination light is enterable from only one direction and the number of PBS steps can be halved. Figures 2(c) and 2(d) show its composition. The limitation of the horizontal number of PBSs by the optical systems in Figs. 2(b)–2(d), which is also drawn as graphs in Fig. 3, is expressed as 2lD because there exist two PBSs in the horizontal direction for every one step of PBSs in the depth direction. The light and dark gray shaded areas in Fig. 3 show feasible regions of the horizontal number of PBSs in this instance.

Experimental setup

We constructed a device with the parameters shown in Table 1. We used a combination lens as to reduce optical distortions. The focal lengths of the individual lenses and in are denoted by f11 and f12, respectively. This device, which uses an optical system and 16 SLMs, corresponds to an SLM with approximately 133 megapixels. We used AM type 4K×2K-pixel reflective liquid crystal on silicon (LCOS) SLMs33,34 that are manufactured by JVC KENWOOD Corporation. Figures 1(c)–1(e) show the experimental setup. We performed the following experiments using hologram data created by computer-generated holography (CGH) technology. Note that in this device, the horizontal and vertical gaps between adjacent SLMs and are 1.8 times the image size of each SLM and due to physical constraints of the SLMs. In other words, for a display area of 1 for the image part, the area of the missing segments is approximately 4.5.

Experiment testing image-readout optical system

We operated the rotatable half-wave plates (HWPs), which are installed on light paths from every colour laser (cf. Figs. 1(a) and 2(d)), to control the ratio of the s- and p-polarized light that is incident on the PBSs. We defined the rotation angles of the HWPs ψR, ψG and ψB to be 0 degrees when the illuminated SLMs of Ce are brightest and the illuminated SLMs of Co are darkest. When the HWPs are rotated until Ce and Co have equivalent brightness, ψR, ψG and ψB become 24, 20 and 23 degrees, respectively. Furthermore, when they are rotated until the Ce are darkest and Co are brightest, ψR, ψG and ψB become 45, 44 and 45 degrees, respectively. Figure 4 shows the experimental results in each colour channel. From these results, we confirmed that the devised optical system is able to illuminate the SLMs from a single direction, the number of PBS steps can be halved and the intended function of the image-readout is achieved.

Figure 4
figure 4

Control of incident light distributions using rotatable HWPs in each colour channel.

The illuminated SLMs of Ce (i.e., C2 and C4) are brightest and the illuminated SLMs of Co (i.e., C1 and C3) are darkest and the rotation angles ψR, ψG and ψB of the HWPs are defined as 0 degrees (a); the HWPs are rotated until Ce and Co have equivalent brightness (b); and they are rotated until the Ce are darkest and Co are brightest (c).

Full size image

Experiment displaying 3D images

We confirmed that objects with different depths could be reconstructed in colour by using time-division multiplexing. Figures 5 and 6 show colour 3D images that are displayed by this system. Figures 5(a) and 6(a) show the design values of the displayed objects. The displayed hologram data was calculated by using the ray-sampling (RS) plane approach35. In this approach, the RS plane is virtually set near the object and the dense ray information is sampled and converted into the wavefront on that plane as the Ray-Wavefront conversion process36. The light field on the hologram plane is then obtained by a two-dimensional (2D) light propagation calculation from the given RS plane to the hologram plane. The dense ray information was rendered as a set of projection images by general computer-graphics (CG) software.

Figure 5
figure 5

Experimental results.

Design values of the displayed objects in the experiment (a). A logotype of NICT is located at 113 mm, the left front wheel of a car is at 118 mm, the left rear wheel is at 143 mm and a Venus statue is at 120 mm behind the hologram plane, respectively. Photographs that were taken from locations directly in front (b), taken by pointing 2.8 degrees right (c) and 2.8 degrees left (d). K.W. prepared hologram data. H.S. took the photographs.

Full size image
Figure 6
figure 6

Experimental results.

Design values of the displayed objects (a). Photographs that were taken of the reconstructed objects focused on the hologram plane (b), focused on the logotype of NICT (c)–(e) and focused on the background object (f)–(h). Photographs (b), (d) and (g) were taken from locations directly in front; (c) and (f) were taken by pointing 2.8 degrees right; and (e) and (h) were taken by pointing 2.8 degrees left. K.W. prepared hologram data. H.S. took the photographs.

Full size image

The photographs in Figs. 5(b), 5(c) and 5(d) were taken from a location directly in front, a location determined by pointing 2.8 degrees right and a location obtained by pointing 2.8 degrees left, respectively. A logotype of NICT is located at 113 mm, the left front wheel of a car is at 118 mm, the left rear wheel is at 143 mm and a Venus statue is at 120 mm behind the hologram plane, respectively.

The photograph in Fig. 6(b) was taken of the reconstructed objects focused on the hologram plane, the photographs in Figs. 6(c)–6(e) were focused on the logotype of NICT and the photographs in Figs. 6(f)–6(h) were focused on the background object. Figs. 6(b), 6(d) and 6(g) were taken from locations directly in front; Figs. 6(c) and 6(f) were taken by pointing 2.8 degrees right; and Figs. 6(e) and 6(h) were taken by pointing 2.8 degrees left. The logotype of NICT is located at 100 mm and the background object is 200 mm behind the hologram plane. When the objects can be reconstructed at different depths, the background object should be blurry in Figs. 6(c)–6(e) and the logotype of NICT should be blurry in Figs. 6(f)–6(h). From the experimental results and the description above, it is apparent that objects with different depths can be reconstructed.

The photographs in Figs. 5 and 6 also show the views of the same displayed objects from other angles. It is apparent that the reconstructed objects can be observed in the designed viewing zone and missing segments that can be seen without the optical system are eliminated.

We also confirmed that a video could be reproduced. Figure 7 shows the experimental results. We used CGH technology to create hologram data for the earth and letters. The letters circle around the earth. The logotype of NICT is located at 104 mm and the centre of the earth is 127 mm behind the hologram plane. This experiment confirmed that this video sequence could be converted to a frame rate with one set of red, green and blue per frame in time-division colour and played smoothly at a 20 fps (frames per second).

Figure 7
figure 7

Experimental results.

Design values of the displayed objects (a) and reconstructed 3D image at video rate (b). The centre of the earth is 27 mm behind the RS plane. K.W. prepared hologram data. H.S. took the photograph.

Full size image

The displayed size of these 3D images is 74 mm × 42 mm (i.e., the diagonal is 85 mm) and the viewing-zone angles are 5.6 degrees horizontally and 2.8 degrees vertically. From the experiments described above, it is apparent that a colour-moving image of 3D objects having an image size of 16 times that of an SLM can be played with a frame rate of 20 fps using the proposed method.

Figures 5,6,7 show that gaps between the SLMs are eliminated. However, the boundaries of the SLMs are observed. In addition, differences of brightness are observed between each small display area.

Discussion

In this paper, we propose a method of producing larger colour electronic holographic images with a newly devised space-saving image-readout optical system for multiple reflection-type SLMs. We applied the proposed method to 16 SLMs to realize full-parallax video holography at a frame rate of 20 fps with an image size of 16 times that of one SLM and a viewing-zone angle of 5.6 degrees. Although the time-division colourization method had a frame rate of 20 fps because we used 60-fps SLMs, we could also reach 60 fps in colour, for example, if 180-fps SLMs were used.

When observing a 3D image with the focus on a distant location from the hologram plane in the depth direction, the boundaries of the SLMs become diffuse and sparse because these boundaries exist on the hologram plane. Actually, the observed boundaries are sparser in Figs. 6(f)–6(h), which are focused on the background object located far from the hologram plane, than the boundaries in Figs. 6(c)–6(e), which are focused on the logotype of NICT located near the hologram plane.

Differences of brightness exist depending on the difference of contrast between each display area in Figs. 4,5,6,7. Since there is no difference of contrast among SLMs, we think that differences among the optical specifications of the various PBSs that comprise an incident optical system become the dominant influence. Since each PBS provides a different contrast, this produces a detrimental effect for viewing the 3D image. We believe that we can balance the differences of display brightness among the various display areas if we compensate for luminance differences among the hologram data that is displayed on the various SLMs.

Methods

Space-saving image-readout optical system

Figures 2(c) and 2(d) show the space-saving image-readout optical system and its operating principles. Rotatable HWPs are inserted in the light paths from every colour laser because light in which s- and p-polarizations are mixed with the intended ratio is to be incident on the PBSs. We use pairs of PBSs. We denote a PBS and SLM at odd-numbered positions here by Do and Co and at even-numbered positions by De and Ce in a pair of PBSs and their corresponding SLMs, respectively. Do and De are arranged in tandem for the incident light path and a HWP is fixed between these PBSs. The s-polarized component of the incident light is reflected by the mirror (e.g., dielectric multilayer mirror, wire grid, etc.) in Do and illuminates Co. The p-polarized component passes through Do and is converted to s-polarized light by the HWP. This light is incident on and reflected by De so that it then illuminates Ce.

Readout light from each SLM consists of mixed polarization light because the s-polarized incident light changed to the p-polarized component of the readout light and this conversion ratio depends on the displayed value of each pixel. Only the p-polarized component of the readout light passes through the PBS and polarizer and is then output from this image-readout optical system. The s-polarized component from the SLMs is reflected by the PBSs and goes back towards the light source. The light that goes back to Do from De is also converted to p-polarized light by the HWP and goes straight through Do ideally. However, there exists a possibility of crosstalk interference by a small quantity of a residual s-polarized component. The polarizer next to Do cuts off this s-polarized component. Essentially, a polarizer does not have to be set next to De, but we use it for luminance equalization of the Do and De optical outputs. Since the modulation type of the SLMs is AM as mentioned previously, polarizers after the PBSs will not affect the modulated wavefront of holograms.

The loss of light that passes through De is greater than the loss of light that only passes through Do because it takes a longer path and also has to pass through the HWP. There exists a possibility of incomplete conversion in the HWP due to wavelength differences, which also leads to a loss of light in De. To compensate for these effects, rotatable HWPs are installed in the light paths from every colour laser as mentioned above. These HWPs can be rotated to control the ratio of the s- and p-components of the incident light and to compensate for luminance differences in the Do and De optical outputs of every colour. Essentially, rotatable HWPs do not have to be set if the corresponding laser equipment is rotatable around each optical axis and fixable at intended attitudes, but we use them to facilitate adjustments of polarizations. The optical systems in Fig. 1 utilize this method.

Image size and viewing-zone angle

If we denote the number of SLM pixels here by Nx and Ny and the pixel pitch in the horizontal and vertical directions by px and py, respectively, then the image sizes Sx and Sy and the horizontal and vertical viewing-zone angles θx and θy are given as follows:

where the horizontal and vertical angles of the first-order diffraction beam θ1x and θ1y are given by equations (5) and (6). With the space-division multiplexing method, if we denote the number of pixels in the horizontal and vertical directions of each SLM by and and denote the number of SLMs by Kx and Ky, then and . The only difference between the horizontal and vertical directions is the viewing-zone angles θx and θy.

Light source optical system and colourization

Figure 1(a) shows the light source optical system that implements colourization based on time-division multiplexing23,24. This is also known as the frame sequential method. This system consists of three lasers (i.e., red, green and blue), rotatable HWPs, wheel shutters, spatial filters and a trichroic prism assembly. The shutters are rotating wheels driven by motors that are synchronized with the SLMs and controlled so that a beam of one of the three colours is selected to pass through the shutter every 1/60 second. After that, the laser beams pass through a spatial filter (i.e., convex lens and pinhole) so that the shape of the beams gradually becomes wider. Three beams are combined into one by a Phillips type trichroic prism assembly. Finally, the combined white beam is collimated by a large size convex lens L0 to illuminate all of the SLMs.

Gap reduction optical system

Figure 1(b) shows the image-readout and gap reduction optical systems. Emitted light from the image-readout optical system is incident on lenses , spatial filters Fm and lenses 6. Spatial filter Fm is located on the common focal plane P1 of lenses and and the focal length f1 of and focal length f2 of are related as follows:

depending on the image size S′ of each SLM and the gap T′ between SLMs. Also, we set the physical path length a and the geometrical path length a′, which was optically compensated by the refractive index of the PBS, between Cm and lenses as shown in equation (1). This enlarges the image due to Cm by (S′ + T′)/S′ and shifts it to (P2).

The missing image segments due to the gaps between the SLMs will disappear. The group of lenses and forms the lens arrays, respectively. Fm is a spatial filter array for eliminating the conjugate beam, carrier beam and high-order beams, which are unnecessary light37. Finally, the light is incident on lens L3 and lens L4. By setting the focal length f3 of L3 and focal length f4 of L4 as follows:

the image that had been enlarged at P2 will be returned to its original size at plane P3.

This optical system can exhibit parallax in both the horizontal and vertical directions. However, as shown in equations (10) and (11), the horizontal viewing-zone angle θx is two times the viewingzone angle of the first-order diffraction beam θ1x, but the vertical viewing-zone angle θy equals θ1y. The reason for this is that spatial filters Fm eliminate the first-order diffraction beam in the lower side of the vertical direction to eliminate the conjugate image, i.e., single-sideband holography38 with half-zone-plate processing is performed for the hologram data that is displayed on the SLM39. We used spatial filters Fm as shown in Fig. 1(b). A mask (gray square with white rectangle aperture) shows a plane figure of the configuration of the spatial filter in the vertical direction. The aperture sizes in horizontal and vertical directions of spatial filters ξF and ηF are given as follows:

The conjugate beam and carrier beam are eliminated by the dashed lined rectangle part in the masked area and the other masked area eliminates the high-order beams. Only the object beam passes through the aperture selectively.