Introduction

Mid-air imaging is a technology that can display real-life images, which can be seen by the naked eye in a physical space; it can be used for the realization of mixed reality. Studies on mid-air images have proposed a system [1,2,3] using aerial imaging by retroreflection (AIRR) structure [4]. There have also been studies on interactive systems such as MARIO [5], HaptoMime [6], and HaptoClone [7], which use a micro-mirror array plate (MMAP) [8]. Research to improve image quality using MMAP includes studies on diminishing images that are not mid-air images [9, 10] and research to widen the viewing angle [11]. An MMAP can form mid-air images using a simple structure. However, the MMAP also generates images of unwanted light. While the position of the mid-air image can be easily calculated based on the properties of the MMAP, the position of the unwanted light has not been formulated, because it depends on the angle and the position of the viewpoint, MMAP, and light source. Therefore, when designing home appliances and interfaces that consider the visibility of mid-air images and images of unwanted light, many parameters such as the placement of optical elements and displays need to be considered. An exhaustive parameter search that is performed by assembling the device is time-consuming and expensive.

Thus, optical simulators are very useful for optical specialists to design mid-air images. For example, using simulation, it is possible to evaluate the design, compare it with the existing design, and find improvements without assembling the optical system. A method of simulating mid-air images involves visualizing light rays [12,13,14]. Considering this method, we can verify the position where the mid-air image is formed and the direction of the rays. However, this method requires specialized knowledge of the mid-air image and each optical element, making it difficult for designers, who are not optical specialists, to utilize it when designing devices.

By simulating the appearance of the displayed mid-air image and the image of the unwanted light, it is possible to evaluate the mid-air image based on how it is perceived by the observer. This can assist optical non-specialists (such as project designers) in designing mid-air imaging systems. Kiuchi et al. demonstrated a physically based simulation that can confirm the appearance (such as the position and shape) of the mid-air image and the image of the unwanted light by reproducing the MMAP [15]. In addition, they confirmed that the position of the mid-air image, shape of the image of stray light, and field of view are the same in reality, demonstrating high reproducibility. The image of stray light is a type of unwanted light image that is particularly difficult to locate compared to the image of transmitted light (another type of unwanted light image). However, when designing an optical system using this method, it is necessary to visually check the mid-air image and presence of unwanted light from the rendered image, as shown in Fig. 1. Therefore, it is difficult for optical non-specialists to recognize the issues and discover methods to improve them by adjusting the placement of MMAPs and displays.

Fig. 1
figure 1

Simulated mid-air image and the image of unwanted light (images of stray and transmitted light)

In this study, we simulate reflective mid-air imaging optical systems that use building materials installed in physical surroundings as the reflective elements. Essentially, we focus on the aforementioned optical systems, because we expect them to promote the integration of real-world and mid-air images, whereby the generated mid-air images are displayed over the surrounding physical backgrounds. The following examples are specifically related to the objectives of this study. Deco-wall [16] is a device that displays a mid-air image in front of an observer by reflecting light on a glossy material. Moreover, Fairlift [17] uses water surfaces as the display surface. Furthermore, EnchanTable [18] is a device that can be retrofitted to a table, and it uses the table as the display surface. Contrary to the aforementioned systems, PortOn [19] realized a portable mid-air imaging device using an imaginary mirror image.

Our proposed method can be useful in enhancing the performance of such reflective mid-air imaging optical systems and in developing their associated applications. This is because the appearance of the generated mid-air images can be designed along with the display space while considering the way in which such mid-air images are displayed. In this study, we propose three methods: a method for extracting mid-air images and images of stray light from computer graphics (CG) rendered images, a method for calculating the visible range of the generated mid-air images, and an approach for evaluating the design parameters of our proposed model. The mid-air image and the image of the stray light are extracted by considering the differences of the images rendered using different bounces at each camera position, after which the visible range of the generated mid-air image is calculated. In addition, other parameters, such as the distance between the MMAP and the generated mid-air image, are adjusted in detail, and they are evaluated by considering the visible range of the obtained mid-air image. Moreover, to demonstrate the effectiveness of our proposed approach, we improve the design of EnchanTable to ensure the elimination of images of stray light. Using our proposed approach, we also compared the design of the novel optical system to that of the existing system (PortOn), and we reduced the size of the system without reducing the visible range of the generated mid-air images.

Procedure

A reflective mid-air imaging optical system using an MMAP was simulated through the ray-tracing method. Blender (version 2.80), which is a popular open-source three-dimensional (3D) software, was used as the simulator. We confirmed that our proposed approach works with version 3.0 of Blender, which was released on December 3, 2021. We also used Cycles, which was built into Blender, as the rendering engine.

Using the relationship between the number of light reflections and the generated mid-air images, the stray light, and the transmitted light, each image is extracted through image processing while changing the maximum number of times by which the ray is tracked. OpenCV version 3.4.5 was used for image processing.

Relationship between the number of times light is reflected and the image

The MMAP used in this study comprises two layers of slit mirror arrays (SMAs). Each layer is placed vertically in overlapping positions. The light incident on the MMAP is reflected by the SMA of each layer, and the mid-air image is generated at the position of the plane symmetry to the MMAP.

Depending on the angle of incident light, the number of times the light is reflected in each layer differs. Based on this number, the light is classified as mid-air imaging, stray, or transmitted light. In this study, transmitted light is defined as the light that passes through the SMAs without being reflected in the MMAP. As shown in Table 1, Maekawa et al. [20] demonstrated the double-reflection mode in which the light reflected in each layer yields mid-air imaging light, the single-reflection mode in which the light reflected in one layer yields stray light, and the non-reflection mode, which yields transmitted light. Particularly, when the angle of incidence is insignificant, it is difficult for light to be reflected sufficiently, and as a result, images of stray light easily appear.

Table 1 Relationship between the number of times light is reflected and the image

Figure 2 shows the position of the mid-air image generated using the light source and the MMAP. As shown in the figure, depending on the angle of the incidence of light, the number of times it is reflected inside the MMAP changes, and a different image, i.e., an image of unwanted light, such as an image of stray or transmitted light, is formed. In the simulator, a rectangle that resembles a monitor is used as the light source, because monitors are often placed as light sources.

Fig. 2
figure 2

Image formation of mid-air images using an MMAP. (a) Image formation of mid-air images; (b) enlarged view of the MMAP. Mid-air imaging light in double-reflection mode. Stray light in single-reflection mode. Transmitted light in non-reflection mode

For simulation, we used the model developed by Kiuchi et al. [15], which satisfied the required functionality by reproducing the appearance of mid-air images, unwanted light, and the locations in which they are displayed with high accuracy. The model comprises SMAs with a thickness of 1.25 mm spaced 0.5 mm apart, followed by a glass object that simply covers the SMAs. The mirrors are made of Blender’s principled BSDF with a metallic value of 0.87 and a roughness value of 0.01, and the glass object is made of glass BSDF with a refractive index of 1.52.

Setting bounces in the ray-tracing method

The key technique employed in this study involves separating the generated mid-air image, the image of stray light, and the image of transmitted light when rendering the mid-air image using the ray-tracing method. This is achieved by generating multiple rendered images with different ray reflections and by considering the differences between these images. Ray tracing is a method for rendering CG images that involves tracking the rays from a camera viewpoint to objects in a scene, including object surfaces, participating mediums, and light sources. The computation procedure is demonstrated as follows: A ray is launched from the viewpoint of each pixel of an image. The subsequent step involves seeking the object surface that intersects with the ray and calculating the contributions of indirect illumination from other objects in the scene as well as direct illumination. This search is performed iteratively until the ray intersects with a light source or until the number of ray bounces exceeds the number of maximum bounces. This is one of the set values. Finally, the pixel is filled depending on the calculated contribution at a surface.

In Blender, we used the ability to set bounces, which is the maximum number of times the tracking path changes direction when it hits a surface. For example, when considering rendering an object with a light source, an object placed on the floor, and a camera, if the number of bounces is 0, the object cannot be rendered, because tracking ends when the tracking path from the camera hits the object. If the number of bounces is increased to 1, up to 1 reflection is calculated. Therefore, the object can be rendered when the tracking path is reflected by the object and when it reaches the light source. If the number of bounces is increased to 2, up to 2 reflections are calculated. Therefore, when the camera starts tracking, it will first reflect on the floor, then on the object, and finally on the light source. As a result, we can render the reflection of the object. Essentially, by increasing the number of bounces, it is possible to calculate repeated reflections of light, thereby expanding the range of objects that can be rendered. By changing the number of bounces using this function, depending on the relationship between the numbers of times light is reflected on the rendered image, it is possible to adjust the amount of light in the rendered image. In the following section, we explain the way in which an image is obtained using only an image of stray light and a mid-air image by setting the number of bounces.

Relationship between the number of bounces and the rendered image

In the simulation of the reflective mid-air imaging optical system, the image with only an image of stray light is obtained by removing the image of the transmitted light from the image rendered using the number of bounces required to observe the image of stray light. The number of bounces required to observe the image is equal to the number of reflections required to render the image. This number increases depending on the image of transmitted light, that of the stray light, and the mid-air image. This is because of the difference in the number of light reflections required in the MMAP. Consequently, in images rendered using the number of bounces required to observe the image of stray light, the image of transmitted light is also observed simultaneously, and it must be removed. Regarding a reflective mid-air imaging optical system, a mid-air image and an image of stray light are observed only when there is a reflective surface, whereas an image of transmitted light is observed regardless of the presence or absence of a reflective surface. This indicates that images of transmitted light can be removed by considering the differences between the rendered images with and without the presence of a reflective surface. Therefore, by setting the number of bounces required to observe the image of stray light and by removing the image of transmitted light, it is possible to only obtain an image of stray light.

A pure mid-air image is obtained by considering the difference between the image rendered using the number of bounces required to observe the mid-air image and the image rendered using the number of bounces required to observe the image of stray light. Regarding the image rendered using the number of bounces required to observe the mid-air image, the image of transmitted light and the image of stray light are also observed simultaneously, and they must be removed. Therefore, as shown in Fig. 3, two images, i.e., the image rendered using the number of bounces required to observe the mid-air image and the image rendered using the number of bounces required to observe the image of stray light, are used. Because Fig. 3(c) is obtained from Figs. 3(a) and 3(b), the image involving a pure mid-air image can be obtained by considering the difference of each rendered image to eliminate the images of stray and transmitted light.

Fig. 3
figure 3

Method for obtaining a pure mid-air image. (a) Image rendered using the minimum number of bounces required to observe a mid-air image; (b) image rendered using the minimum number of bounces required to observe the image of stray light; (c) obtaining a pure mid-air image by considering the differences between (a) and (b)

Region extraction through image processing

In this section, we describe the image processing procedures for extracting a pure mid-air image and the images of stray light portions from the rendered image. An image with a black background and a total of eight white boxes at the four vertices of the rectangle as well as the midpoints of the four sides was used as a marker, after which rendered images were obtained. This was done to prevent the mid-air image from being lost owing to the overlap of the image of stray light and the mid-air image when calculating their differences during image processing and to obtain a pure mid-air image correctly. Light attenuation resulting from the increase in the number of reflections in each layer of the MMAP and light occlusion caused by other objects are considered as the conditions for the missing mid-air image. In addition, when obtaining a pure mid-air image using the approach described in the previous section, the overlap of the mid-air image and the image of stray light may result in a part of the mid-air image being removed along with the image of stray light, thereby making it impossible to obtain a correct image. Therefore, an image with eight white boxes placed at each vertex and the midpoint of each edge are used to prevent the mid-air image from being removed when mid-air images and images of stray light overlap and to recognize instances in which the mid-air image is missing.

The image of stray light and the mid-air image were extracted from the images obtained through the procedure described in Sect. 2.3 by counting the number of white boxes using OpenCV. As demonstrated in Sect. 2.3, by applying a Gaussian blur to the images of stray light or mid-air images and binarizing them, it is possible to correct and extract each image regardless of whether it is missing as a result of the overlap of the mid-air image and the image of stray light when subtracting such images. This operation is useful in the effective detection of marker by correcting for cases in which pixels that should be drawn as white are drawn as black owing to a small number of samples. In the ray tracing method, the number of tracking paths is determined by the sampling rate. Therefore, because this number is small and there are insufficient tracking paths to render the color of the light source in one part of the image, pixels can be rendered as black. Thereafter, the number of white boxes present in the image is measured by detecting the contours of the white boxes using the findContours function. It is necessary to extract a collection of points when checking the number of white boxes. Contour extraction is the standard method when this task is performed through image processing. Considering the image of stray light, it is determined to be present when there is more than one white box. Regarding mid-air images, they are considered present when there are eight white boxes (Fig. 4).

Fig. 4
figure 4

Procedure for detecting marker. (a) Pure mid-air image obtained by removing stray light using the method described in Sect. 2.3; (b) application of Gaussian blur; (c) binarization; (d) marker detection

Evaluation of the reflective mid-air imaging optical system

Improvement of the design of the existing optical system

Using the method presented earlier, we attempt to improve the design of EnchanTable by moving the MMAP away from the bottom of the device to eliminate the image of stray light, as shown in Fig. 5. As mentioned earlier, EnchanTable is a reflective mid-air imaging optical system. Considering this optical system, the light emitted from the display directly enters the MMAP, reflects off the glossy surface, and generates a mid-air image. However, there still exist images of stray light that are observable from different angles. Therefore, the design has been improved.

Fig. 5
figure 5

Angle of incidence adjusted by lifting the MMAP from the bottom of the device. The image of stray light is eliminated by controlling the angle of incidence

When light enters the MMAP, an image of stray light is generated if the angle of incidence is small, as shown in \(\theta _{1}\) in Fig. 5. Therefore, we considered removing the image of stray light by moving the MMAP away from the lower part of the device and setting the angle of incidence that was less likely to result in the image of stray light, such as \(\theta _{2}\). When light enters the MMAP from an angle close to a right angle, it is either transmitted without being reflected in the MMAP or it leaves the MMAP without being reflected in the other layer after being reflected in one layer, thereby resulting in a stray light. Therefore, the angle of incidence was adjusted to reflect the light once by each layer and to generate a mid-air image.

The image of stray light was extracted at each camera position in each design and at a range in which a complete mid-air image could be observed without the image of stray light (full visible range). We investigated a design in which the distance \(x_d\) between the MMAP and the mid-air image and the distance \(h_{up}\) from the bottom of the MMAP were changed. The \(x_d\) range was set to 13 conditions of 100–400 mm at 25 mm intervals, and the \(h_{up}\) range was set to 16 conditions of 0–300 mm at 20 mm intervals for each \(x_d\). The size of the MMAP was \(H_{m} = 488\) mm, the horizontal distance between the mid-air image and the camera was \(D_{c} = 500\) mm, and the size of the display was \(H_{d} = 175\) mm. The range of the camera position in the simulation was defined as the range (simulation range) in which the complete mid-air image was visible in the angle of view of the camera. This range is a rectangle with a height of \(h_{min}\) to \(h_{max}\) and a width of \(w_{max}\), as shown in Fig. 6. Because the simulation range is the camera’s position in which the line of the camera’s angle of view and the edge point of the mid-air image intersect, it can be calculated as follows:

$$\begin{aligned} h_{min} = {} (H_{d}+h_{up})\frac{x_d+D_{c}}{x_d}-h_{up} \end{aligned}$$
(1)
$$\begin{aligned} h_{max} = {} (h_{up}+H_{m})\frac{D_{c}}{x_d} \end{aligned}$$
(2)
$$\begin{aligned} w_{max} = {} \frac{H_{m}-H_d}{2}\frac{x_d+D_{c}}{x_d}-\frac{H_{m}}{2}. \end{aligned}$$
(3)

The full visible range was obtained by extracting the image of stray light from images rendered within the simulation range and by removing the range in which the image of stray light was visible from the simulation range.

Fig. 6
figure 6

Geometric arrangement of the simulation settings. a Side view; b top view

The change in the full visible range achieved by moving the MMAP away from the bottom of the device is shown in Fig. 7, and the full visible range is maximized when \(h_{up}=160\) mm or more. The horizontal and vertical axes represent \(h_{up}\) and the ratio of the full visible range to the simulation range, respectively. Particularly, we determined that a significant improvement can be achieved by increasing \(h_{up}\) at \(x_{d} = 400\) mm.

Fig. 7
figure 7

Percentage of the full visible range. The range is the largest at \(h_{up} = 160\) mm regardless of the value of \(x_d\)

The full visible range in each \(h_{up}\) can be used as a reference when designing the intended use and user. Figure 8 shows the full visible range in the simulation range. Considering the figure, it can be observed that by increasing \(h_{up}\), the full visible range becomes wider, and it moves upward simultaneously. Therefore, the location of the device can be determined by considering the user’s eye position during interaction. For example, Fig. 8(b) shows that by moving the MMAP 20 cm away from the bottom of the device when the mid-air image is placed 20 cm in front of the device, it is possible to provide a mid-air image that can be viewed standing up by a person with an eye height of 110 cm to 175 cm.

Fig. 8
figure 8

Full visible range at different \(h_{up}\) for each \(x_d\). (a) \(x_d = 100\) mm; (b) \(x_d = 200\) mm; (c) \(x_d = 300\) mm; (d) \(x_d = 400\) mm. The full visible range moves upward as \(h_{up}\) increases

Figures 7 and 10 show the results of the comparison between mid-air images displayed using the assembled device, as shown in Fig. 9. We set \(x_{d} = 300\) mm, \(D_c = 500\) mm, and \(h_{up} = 0\) mm, 100, and 200 mm in three cases for each configuration. The height of the camera from the glossy surface was set at 500, 650, and 800 mm for \(h_{up} = 0\) mm; 650, 800, and 950 mm for \(h_{up} = 100\) mm; and 800, 950, and 1100 mm for \(h_{up} = 200\) mm. A total of nine points were considered for each condition. As shown in Fig. 10, this was done to compare the full visible range obtained through the simulation with the actual full visible range. The results of the experiment using the actual device confirmed that the mid-air image could be observed with no stray light within the full visible range and stray light was observed outside the range.

Fig. 9
figure 9

View of EnchanTable used in the experiment

Fig. 10
figure 10

View of the mid-air image at \(x_{d} = 300\) mm. (a) \(h_{up} = 0\) mm; (b) \(h_{up} = 100\) mm; (c) \(h_{up} = 200\) mm

Designing a new optical system

Using the aforementioned method, a new optical system, PicPop [21], which is shown in Fig. 11, is compared with the existing optical system (PortOn). A design was sought to reduce the size of the device without reducing the range in which the entire mid-air image was observed. As mentioned earlier, PortOn is a reflective mid-air imaging optical system [19]. Considering this optical system, the light emitted from the display is reflected by a mirror, and it passes through the MMAP, where it is reflected on a glossy surface, thereby generating a mid-air image. By placing the polarizer behind the MMAP, unwanted light that would otherwise go out into the ground is eliminated. PicPop is an optical system created to realize a pop-up picture book using mid-air images. However, the design can be improved. It was created based on PortOn and Osato et al.’s study [22] on device miniaturization. The volume of the device was reduced by tilting the mirror inside it and adjusting the display position. The image of unwanted light was removed using the louver films in front of the MMAP (louver film 1) and display (louver film 2). We confirm that the luminance can be improved compared to that of PortOn using a polarizer.

Fig. 11
figure 11

a Design of PicPop. The intersection point of the MMAP and reflective surface is the origin O. A: Top of display, B: bottom of display, and C: top of mirror; b structure of the louver film

The mid-air image was extracted at each camera position in each design, and the range in which the entire mid-air image was observed (visible range of the mid-air image) was obtained. Here, we investigated a design in which the distance \(x_d\) between the MMAP and the mid-air image, the angle \(\theta\) of the mirror, and the combinations of the louver films that control the direction of the image of transmitted light rays were altered. The distance \(x_d\) was investigated based on five conditions, every 25 mm from 150 to 250 mm. The size of the MMAP was \(H_m = 220\) mm, the horizontal distance between the mid-air image and the camera was \(D_c = 300\) mm, and the size of the display was \(H_d = 100\) mm. The simulation range was determined using the same equations as those presented in 3.1. Note that \(h_{up}= 0\). The visible range of the mid-air image was defined as the range in which the mid-air image could be extracted from the image rendered within this simulation range. Contrary to the full visible range, this range includes the range in which the image of stray light is visible. This is because we only focus on the size of the optical system.

\(\theta\) was investigated for every \(1^\circ\) in the range of \(0^\circ \le \theta \le 45^\circ\) at a value that could be physically assembled. In this study, the case in which the display does not protrude from the MMAP to the mid-air image side was defined as a design that could be physically assembled. When the intersection of the MMAP and the reflective surface represents the origin, the three points shown in Fig. 11(a), the upper part of display A (\(A_x\), \(A_y\)), the lower part of display B (\(B_x\), \(B_y\)), and the upper part of mirror C (\(C_x\), \(C_y\)) can be obtained using the following equation:

$$\begin{aligned} A_{y} = H_{d}cos2\theta + x_{d}sin2\theta , \ \end{aligned}$$
(4)
$$\begin{aligned} B_{x} = x_{d}cos2\theta , \ \end{aligned}$$
(5)
$$\begin{aligned} C_{x} = \frac{x_{d}H_{m}}{H_{m} + x_{d}tan\theta }. \ \end{aligned}$$
(6)

Considering these parameters, the ratio of the device volume of PicPop to the device volume of PortOn was calculated using the following equation:

$$\begin{aligned} \text{ Volume } \text{ ratio } = ( max( A_{y},H_{m} ) ) \times \frac{max(B_{x},C_{x})}{ H_{m} \times x_{d}}. \end{aligned}$$
(7)

The value is set to zero if assembly is not possible. Figure 12 shows the ratio of the device volume of PicPop to the device volume of PortOn when \(\theta\) is varied at each \(x_{d}\). As shown in the figure, the larger the value of \(\theta\), the smaller is the device.

Fig. 12
figure 12

Relationship between the degree of mirror tilt of PicPop and the ratio of the device volume for PortOn. The volume of PicPop becomes smaller than that of PortOn by increasing the mirror tilt \(\theta\). \(\theta\) was investigated for every \(1^\circ\) in the range of \(0^\circ \le \theta \le 45^\circ\) at a value that could allow physical assembly

Regarding the louver film used in PicPop, five patterns were investigated: four patterns using two types of existing films combined in front of the MMAP (louver film 1) and the display (louver film 2) and one pattern using a calculated louver film. As shown in Fig. 11(b), the louver film controls the direction of the transmitted rays, and it comprises multiple light-shielding layers arranged at the same angle. As indicated by the crosses in Fig. 11(a), louver film 1 eliminates the transmitted light that reaches the eyes directly, whereas louver film 2 eliminates the light that enters the MMAP directly from the display. The characteristics are determined by the viewing angle \(\alpha\), the louver angle \(\beta\), and the distance p between each light-shielding layer. In this study, we used two types of existing films with \(\beta = 25^\circ\), \(p = 0.132\) mm, \(\alpha = 48^\circ\), and \(60^\circ\).

For the calculated louver film, the film in front of the MMAP was designed to ensure that the light emitted from the MMAP would form a mid-air image from any point and be blocked otherwise; the film in front of the display was designed to ensure that the light from the display would enter the MMAP from any point and be blocked otherwise. With the existing films, the visible range of mid-air images is significantly reduced depending on the value of \(x_d\). To solve this problem, we used a calculated film to verify whether the visible range of the mid-air image could be maintained regardless of the value of \(x_d\). We use films with \(p=0.132\) mm, which is the angle shown in the following equation when the maximum incident angle of light that can be transmitted through the louver film is \(\phi _{max}\), and the minimum incident angle is \(\phi _{min}\) for \(\alpha\) and \(\beta\)

$$\begin{aligned} \alpha= {} \phi _{max} - \phi _{min} \end{aligned}$$
(8)
$$\begin{aligned} \beta = {} \arctan \left( \frac{\tan \left( \phi _{max} \right) +\tan \left( \phi _{min} \right) }{2}\right) . \end{aligned}$$
(9)

The viewing angle of the louver film in front of the MMAP (louver film 1) is denoted as \(\theta _{MMAP}\) and the one in front of the display (louver film 2) is denoted as \(\theta _{display}\). The combination of the two in PicPop is (\(\theta _{MMAP}\), \(\theta _{display}\)). PortOn was used as the louver film only in front of the MMAP. In the simulation, we used a model that reproduced the louver film by placing slits at intervals.

Figure 13 shows the visible range of the mid-air images of PortOn and PicPop at \(\theta = 20^ \circ\) for each louver film combination for each value of \(\theta _{MMAP}\). It was determined that PicPop maintained more than 85% of the visible range of the mid-air image of PortOn, except in the case of the louver film combination (60, 48). In the case of (60, 48), measurements less than 200 mm were below the 85% line. This ratio was smaller for the (60, 48) combination. This may be because more light is blocked by the louver film in front of the display than in front of the MMAP. Particularly, when the viewing angle of the louver film installed in front of the MMAP (louver film 1) was \(48^\circ\), the visible range of the mid-air image was the same as that of PortOn, and the system volume could be reduced by more than 20%.

Fig. 13
figure 13

a Ratio of the visible range of the mid-air image of PicPop to that of PortOn with regard to \(\theta _{MMAP} = 48^\circ\); b ratio of the visible range of the mid-air image of PicPop to that of PortOn when \(\theta _{MMAP} = 60^\circ\). Considering most of the louver film combinations, PicPop maintained more than 85% of the range achieved using PortOn

Figure 14 shows the ratio of the visible image of the mid-air range to the simulation range when \(\theta \ge 20^ circ\) and \(\theta _{MMAP} = 48^\circ\) for each louver film combination. It can be observed that the percentage is always above 85% when the calculated louver film is used, and the visible range of the mid-air image is more stable than when the existing film is used, regardless of the value of \(x_d\). This is because when the value of \(x_d\) is small, the width of the light emission angle from the light source to the MMAP is large, and the visible range of the mid-air image is particularly susceptible to the shading of the louver film. The calculated louver film does not shade the light in the wide range of the emission angle. Therefore, the mid-air image can be observed in a wider range than when using the existing louver film. Contrarily, when the value of \(x_d\) is large, the visible range of the mid-air image is not significantly affected by the shading of the louver film, because the width of the angle of emission of light from the light source to the MMAP is small.

Fig. 14
figure 14

Ratio of the visible range of the mid-air image to the simulation range for each louver film combination with \(\theta = 20^\circ\) and \(\theta _{MMAP} = 48^\circ\)

Figures 16 and 17 show the results of the comparison of the mid-air images displayed by the assembled device, as shown in Fig. 15. We set \(x_{d} = 200\) mm, \(\theta = 20^\circ\), and \(D_c = 300\) mm. Images were taken at three camera heights of 200, 250, and 300 mm from the reflective surface and three camera widths of 0, 75, and 90 mm when the front of the mid-air image was set to zero for nine points. As shown in Fig. 16, this approach was aimed at comparing the visible range of the mid-air image obtained through the simulation with the actual visible range of the mid-air image. The results of the experiment using the actual device confirmed that the mid-air image could be observed without missing within the visible range of the mid-air image. Furthermore, a part of the mid-air image was missing outside the range.

Fig. 15
figure 15

View of the actual assembled device. a PortOn; b PicPop

Fig. 16
figure 16

Visible range of the generated mid-air image considering \(\theta _{MMAP}=48^\circ\). a PicPop and b PortOn

Fig. 17
figure 17

View of the generated mid-air image. a PicPop and b PortOn

Conclusion

We proposed methods for generating mid-air images by combining images rendered under multiple numbers of bounces in appearance-based simulations and for evaluating reflective mid-air imaging systems from the full visible range and that of the generated mid-air image. Using this approach, we improved the design of existing optical systems to facilitate the elimination of the image of stray light. In addition, to reduce the size of the equipment, design improvements were made by comparing the new optical system to the existing optical system.

This method can be used in actual reflective mid-air imaging systems, and it is useful in studies associated with the design improvement and the related applications of mid-air imaging optical systems. Improvements can be made in the design of optical systems to ensure that such systems display mid-air images with conditions, such as the image being observable over a wide range and only in a specific range, and without visible images of stray light. Moreover, the location of the device and the method for interaction can be considered by visualizing the range in which the mid-air image is observed. Therefore, considering the way in which the mid-air image should be displayed and the way in which the observer should perceive the mid-air image while designing the optical system can help in understanding this interaction, because one can design the display space as well as the appearance of the mid-air image simultaneously.

In this study, the accuracy of the display of the louver film model is low only when the position of the image of stray light or that of the mid-air image is not considered. The following development is required when the luminance or sharpness of the image is considered. First, considering an actual display, the luminance decreases as the angle of light emission decreases, whereas the simulation display model emits light uniformly through \(180^\circ\). Consequently, there are cases in which light that should not reach the viewer reaches the viewer. Second, the actual louver film is filled with silicone rubber between the light-shielding layers, whereas the louver film used in the simulation comprises only the light-shielding layers. Therefore, there is no attenuation of the transmitted rays, and 100% of the light from the light source reaches the viewer within the viewing angle. Thus, by changing the model of the display and the louver film to a model with characteristics that are closer to reality, a highly accurate simulation that includes light attenuation can be performed.