Mobile visualization of density fields using smartphone background-oriented schlieren

Abstract

The conventional background-oriented schlieren (BOS) technique is an image-based technique that can calculate the density field in fluids using two static images [i.e., an undistorted background image (reference image) and a distorted background image due to the density change in fluids (target image)]. This paper proposes the smartphone BOS (SBOS) technique, which offers the measurement of the density gradient using the high-speed imaging feature of the smartphone being carried with a moving observer. The conventional BOS with a fixed camera visualizes the density gradient by comparing the reference image and the target image. In contrast, SBOS can obtain the time difference of the density gradient field. A reference image in SBOS is a target one at a previous time step. The movement of the smartphone is canceled using a registration technique for image accurate alignment. Three demonstrations are conducted to perform SBOS. First, in a static situation, the density field of heated air by a gas burner is visualized by comparing between SBOS and conventional BOS. Second, the local displacement of density field and the error displacement is estimated quantitatively when the smartphone is moving. Third, SBOS using an embossed wallpaper to visualize the density field is performed in the mobile condition. These achievements suggest that SBOS is an effective system to visualize the density field using only the smartphone, and is expected to be a useful tool such as a preliminary experiment in the laboratory and a teaching tool for general smartphone users.

1 Introduction

Non-contact visualization of the density field is important for various fields (e.g., airplane (Swanson and Light 1992), shockwaves (Pandya et al. 2003), among others). The conventional way of visualization of the density field is the schlieren technique (Moore 1980; Settles 2001), which requires the relatively complex optical system fixed in the laboratory. Based on the schlieren technique, the background-oriented schlieren (BOS) technique (Richard and Raffel 2001; Raffel 2015) has been developed. BOS can visualize and quantify the density field using a simple setup (i.e., a random-dot background plate, a camera and a light source, see Fig. 1i). Recently, the BOS has been studied thanks to the recent development of digital cameras. BOS is applicable for compressed air (Venkatakrishnan and Meier 2004), mixed air (Kirmse et al. 2011; Tan et al. 2015) and underwater shock waves (Yamamoto et al. 2015; Hayasaka et al. 2016). More recently, Hargather and Settles (2010) proposed the improved BOS technique to visualize the density field of the shock wave induced by gun firing in the air. The so-called “natural-BOS” method proposed by Hargather and Settles (2010) uses outdoor landscapes instead of a patterned background plate and, thus, requires only a camera.

Based on previous BOS studies, this paper proposes a novel BOS technique using the smartphone [hereafter, the smartphone BOS (SBOS)], which offers the high mobility of the measurement system. The possible applications of the method proposed in this paper are, for example, a preliminary experiment in the laboratory and a teaching tool (Cierpka et al. 2016) among others. This is the first time to incorporate smartphone into BOS method, while the smartphone guarantees the high mobility and frame speed of the camera and has a high market penetration rate. In fact, several researches have shown the applicability of the smartphone to the scientific research (Cierpka et al. 2016; Chen et al. 2017; Marshall et al. 2017; Aguirre-Pablo et al. 2017). Note that the smartphones were fixed in their researches, while the smartphone in SBOS technique proposed in this paper moves with a moving observer. Raffel et al. (2002, 2014) capture the density gradient with two cameras simultaneously without the reference image. In contrast, SBOS obtains the time difference of the local displacement vector by comparing a reference and target images, which were captured by different time and positions. Heineck et al. (2016, 2019) obtain the density gradient of a series image using an unaltered reference image in the situation which the camera and object are moving, while in SBOS measurement a reference image is then updated from the immediately preceding target image. The global displacement caused by the movement of the smartphone is corrected by the registration.

In this contribution, the principle of SBOS is described in Sect. 2.1 and the procedure of image analysis is described in Sect. 2.2. The configuration of the experimental setup is described in Sect. 3. Three demonstrations are conducted to reveal the performance of SBOS. In Sect. 4.1, the density field of the air heated by a gas burner is visualized using SBOS to compare the local displacement between the conventional BOS and SBOS. For direct comparison, the smartphone is set at a static place in this experiment. In Sect. 4.2, the local displacement of the density field and the error displacement is estimated quantitatively in the dynamic situation. In Sect. 4.3, SBOS without using a background plate to visualize the density field is performed in the dynamic situation. Section 5 discusses the conclusion and the future outlook.

2 Smartphone BOS

2.1 Principle of smartphone BOS

Fig. 1

Comparison of (i) conventional BOS (Richard and Raffel 2001; Yamamoto et al. 2015) and (ii) smartphone BOS

In this section, the difference of conventional BOS and SBOS is explained. The schematic of the conventional BOS system is shown in Fig. 1i (Venkatakrishnan and Meier 2004; Yamamoto et al. 2015). The background plate is placed behind the density gradient. The reference image without the density gradient is captured (Fig. 1i above). When the fluid experiences a density change, a ray (red line in Fig. 1i below) reflected from a background plate is refracted by the change in density. The path-integrated quantity appears on the image plane of the camera as the local displacement vector. The two-dimensional local displacement vector ${{{\varvec{u}}}_{t}}$ on the image plane at time t is expressed by the following relation (Venkatakrishnan and Meier 2004):

$$\begin{aligned} {\varvec{u}}_{t} =\frac{ { Z }_\mathrm{D}f }{ { Z }_\mathrm{B} } \frac{ 1 }{ { n }_{ 0 } } \int _{ { Z }_\mathrm{D}-\varDelta { Z }_\mathrm{D} }^{ { Z }_\mathrm{D }+\varDelta { Z }_\mathrm{D} }{ \nabla n \mathrm{d}z }, \end{aligned}$$

(1)

where $Z_\mathrm{D}$ is the distance from the background plate to the density gradient, $Z_\mathrm{B}$ is the distance from the background to the lens, f is the focal length, $n_{0}$ is the refractive index in the air (= 1.000292) and n is the refractive index with the density gradient in fluid. The refractive index n is related with the density $\rho$ through the following equation (Raffel 2015; Merzkirch 2012):

$$\begin{aligned} n=K\rho +1, \end{aligned}$$

(2)

where K is the Gladstone–Dale constant ($= 2.2583\times 10^{-4} \hbox {m}^{3}/\hbox {kg}$). Subscripts 1 and 2 display the condition in the time $t_{1}$ and $t_{2}$ in Fig. 1, respectively. The local displacement is obtained by the cross-correlation method (e.g., PIV) comparing the reference image and the target image. In conventional BOS, the only one reference image without the density gradient is used.

In contrast, SBOS updates the reference image in the time sequence using the high-speed capture mode of the smartphone. This means that the density gradient is captured in both the reference image in time $t_{1}$ (Fig. 1ii above) and the target image in time $t_{2}$ (Fig. 1ii below). A difference of the local displacement vector in the time coordinate ${\varDelta {{\varvec{u}}}_{t}}$ is described as the following equation:

$$\begin{aligned} {\varDelta {{\varvec{u}}}_{t}} = {{\varvec{u}}}_{t2} - {{\varvec{u}}}_{t1}, \end{aligned}$$

(3)

where ${{\varvec{u}}}_{t1}$ and ${{\varvec{u}}}_{t2}$ are the local displacement vectors containing the horizontal component $u_{t,x}$ and the vertical component $u_{t,y}$ in time $t_{1}$ and $t_{2}$, respectively. Clearly, the time difference of local displacement vector ${\varDelta {{\varvec{u}}}_{t}}$ is zero in Eq. 3 if both local displacement vectors are the same. In other words, SBOS can visualize the density gradient of the unsteady flow or the steady flow when the smartphone is moving.

2.2 Image analysis

Fig. 2

Procedure of image analysis. The density gradient flow is recorded by the smartphone in Step 1. Images A, B, C and D are captured at different times and position. The reference image is sequentially updated. The movement of the smartphone is canceled using a registration technique for the accurate alignment of images in Step 2. The local displacement field is obtained using the cross-correlation method in Step 3

The procedure of the image analysis is shown in Fig. 2. All process is executed in Matlab. First, a series of the snapshot is captured by the smartphone (Fig. 2, Step 1). After Step 1 in Fig. 2, the color information which includes the red color intensity R, green color intensity G, and blue color intensity B is converted into gray-scale information (intensity GR) using the following equation:

$$\begin{aligned} GR = 0.2989 \times R + 0.587 \times G + 0.114 \times B \end{aligned}$$

(4)

The converted gray-scale images is used for the registration analysis (Step 2 in Fig. 2) and cross-correlation analysis (Step 3 in Fig. 2). In the conventional BOS, the local displacement is obtained by comparing the reference image without the density gradient and the target image with the density gradient. In a series of snapshots, one reference image is used and is not updated. However, in SBOS, it is assumed that an observer is walking while holding the smartphone. In a series of snapshot captured in the dynamic situation, the location of a background plate is different for each image due to the movement of the smartphone. Here, the local displacement caused by the density gradient cannot be measured since the movement of the camera is added as the local displacement. Then, the motion of the background plate is corrected by aligning the reference and target images as illustrated in Fig. 2, Step 2. The rigid registration is performed by “imregister” Matlab function. The similarity of the intensity distribution between the target image (Fig. 2, image D) and the reference image (Fig. 2, image C) is measured by the Mattes Mutual Information metrics (Mattes et al. 2001). The One Plus One Evolutionary optimizer searches iteratively for the parameter (i.e., the geometric transformation matrix) with highest similarity (Styner et al. 2000). The minimum size of the search radius is $1.5\times 10^{-4}$ [–] which determines the coefficient and the precision. The growth coefficient of the search radius is 1.01 [–] which relates to the coefficient of the conversion. The maximum iteration number of the optimizer is 300.

By multiplying the intensity distribution matrix of the original image by the geometric transformation matrix, the registered image is obtained (Fig. 2, image C’). The geometric transformation used in this study corresponds to the linear transformation such as the translation, the rotation and the scaling. Registration is typically done one time per image. If the background deviation between images after registration is more than 0.25 pixels, registration is performed again.

Next, the local displacement field in the inspection region is calculated by the cross-correlation method (Fig. 2, Step 3) between the registered image and the target image (Fig. 2, image B and image C’). By applying this process to all images, a time history of the local displacement field is obtained. The PIV lab 1.4 as the analysis tool is utilized for the cross-correlation method (Thielicke and Stamhuis 2014). The Fast Fourier Transform (FFT) multi-pass interrogation from 64 to 8 pixels is selected.

3 Experimental setup

Fig. 3

(i) A schematic of experimental setup with a background plate. The inset shows the definition of the moving axis. (ii) Setup for estimating the effect of the natural background on the local displacement

A setup for capturing the density gradient using a background plate is shown in Fig. 3i. The flame of a gas burner is utilized as the origin of the density gradient. The smartphone (iPhone 7®, Apple Inc.) is placed on the movable stage. The human displacement is represented by the movement of the stage. The density gradient is captured in the high-speed-photographing mode or the single-shot mode of the smartphone. In the high-speed-photographing mode, the spatial resolution is $1920 \times 1080$ pixels and the frame speed is 120 fps. In the single-shot mode, the spatial resolution is $4032 \times 3024$ pixels. The focal length f is 4.15 mm. The aperture of iPhone 7 confirmed from the Exif information of the original image is f/1.8. The effect of the lens distortion on BOS analysis is discussed in Appendix. The shutter speed and ISO are automatically adjusted by the smartphone. The autofocus is not used. The focus position is fixed to the background plate before photographing. An indoor white fluorescent lamp installed on the ceiling is used as a light source. A randomly colored background plate is mounted behind the density gradient. The size of the color square of the background plate is $5 \times 5\hbox { mm}^{2}$. The distance $Z_\mathrm{B}$ from the smartphone to the background plate is 1 m, and the distance $Z_\mathrm{D}$ from the density gradient to the background plate is 0.8 m.

A setup shown in Fig. 3ii is utilized to estimate the effect of the natural background on the local displacement. The local displacement of the wall with the embossed paper as a natural background and that of the mosaic-patterned background image projected on the wallpaper are compered. The distance $Z_\mathrm{B}$ from the smartphone to the background plate is 0.6 m, and the distance $Z_\mathrm{D}$ from the density gradient to the background plate is 0.4 m. The data of the projected background image are the same as the background used in Fig. 3ii. Then, the focus position is fixed to the embossed wallpaper before photographing. A background image is projected on the wallpaper using the projector. When evaluating the local displacement by not using a background plate, a projector is turned off.

In addition, for visualizing the density gradient with the steady-phase object, a glass plate ($18 \ (H) \ \mathrm{mm} \times 14 \ (W) \ \mathrm{mm} \times 6 (D) \ \mathrm{mm}$) made from the borosilicate glass is placed at an angle of 30 [$^{\circ }$] to the line of sight of the smartphone. The experimental setup except for the density objet is the same as Fig. 3ii. The distance $Z_\mathrm{D}$ from the glass plate to smartphone is 0.5 m, and the distance $Z_\mathrm{B}$ from the smartphone to the background is 0.7 m. The mosaic patterned background is projected by the projector. The smartphone is placed on a single axis stage and moves horizontally.

4 Result and discussion

4.1 Visualization of density gradient by stationary observer

Fig. 4

(i) An image without the density gradient at time $t_{0}$. (ii) An image with the density gradient at time $t_{1}$. (iii) An image with the density gradient at time $t_{2}$

Fig. 5

Comparison of the local displacement field between conventional BOS (Fig. 5i–iii) and SBOS (Fig. 5iv–vi). The third row shows the difference of the local displacement field (Fig. 5vii–ix). The first column and the second column and the third column show the local displacement in x-coordinate, that in y-coordinate and magnitude, respectively

The temporal change of the local displacement vector ${\varDelta {{\varvec{u}}}_{2,1}}$ between time $t_{1}$ (Fig. 4ii) and time $t_{2}$ (Fig. 4iii) using SBOS and conventional BOS is compared. In SBOS, the local displacement vector ${\varDelta {{\varvec{u}}}_{2,1}}$ is directly obtained using the cross-correlation method. In conventional BOS, an image without the density gradient at time $t_{0}$ (Fig. 4i) is used as a reference image. The local displacement vector ${\varDelta {{\varvec{u}}}_{2,1}}$ is expressed as the difference between ${\varDelta {{\varvec{u}}}_{2,0}}$ and ${\varDelta {{\varvec{u}}}_{1,0}}$ as

$$\begin{aligned} {\varDelta {{\varvec{u}}}_{2,1}} = {\varDelta {{\varvec{u}}}_{2,0}} - {\varDelta {{\varvec{u}}}_{1,0}} \end{aligned}$$

(5)

where ${\varDelta {{\varvec{u}}}_{2,0}}$ is the displacement between Fig. 4iii and i, and ${\varDelta {{\varvec{u}}}_{1,0}}$ is the displacement between Fig. 4ii and i. The local displacement ${\varDelta {{\varvec{u}}}_{2,1}}$ obtained by SBOS and conventional BOS is equivalent theoretically. The comparison of local displacement vector in the horizontal component $\varDelta u_{t,x}$ and the vertical component $\varDelta u_{t,y}$ and magnitude ${\varDelta {{\varvec{u}}}_{t}}' = \sqrt{ {\varDelta u_{t,x}}^{2} + {\varDelta u_{t,y}}^{2}}$ [pixel] in each approach is displayed in Fig. 5. As expected, SBOS and conventional BOS data are in good agreement.

The maximum displacements depend on the spatial resolution of image, the background size, the size of interrogation area in the cross-correlation method, etc. In our experimental condition, the maximum local displacement is about 3 pixels in Fig. 5iv.

The observed phenomenon (i.e., the heated air) is blurry in this optical arrangement even though the smartphone has super wide-angle lens. However, the sensitivity of the BOS measurement is better with large $Z_\mathrm{D}$ (see Eq. 1). In addition, the observed phase-shift object is transparent. Since the aim of this research is the detection of the density gradient, the image blur in this paper is not discussed in detail. If the accurate values of local displacements are needed, the image blur must be considered seriously. For comparison, the local displacement obtained with a scientific camera is discussed in "Appendix".

4.2 Visualization of density gradient by moving observer

4.2.1 Visualizing the unsteady flow with the depth movement of a smartphone

Fig. 6

(i) The time difference of local displacement field (magnitude) obtained by SBOS in the moving situation. The depth movement of the smartphone is 1 mm. (ii) The average local displacement ${\varDelta {{\varvec{u}}}_{tm}}'$ obtained by each column in (i). Red circle and blue circle indicate the maximum average local displacement $\underline{{\varDelta {{\varvec{u}}}_{tm}}'}$ and the average error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ in the first column, respectively. (iii) The maximum average local displacement and the average error displacement in the first column versus the depth movement of the smartphone

When a human moving in the depth direction captures images for SBOS processing, it is expected that the local displacement will change depending on the position of the smartphone in the depth direction from the geometric relationship of BOS (Eq. 1). In Fig. 6, the local displacement field from images captured in the single-shot mode is computed. Figure 6i shows the time difference of local displacement field (magnitude) ${{\varDelta {{\varvec{u}}}_{t}}}'$ caused by the heated air. The distance in the depth direction between the reference image (image C, Fig. 2) and the target image (image D, Fig. 2) is 1 mm in Fig. 6i. Figure 6ii shows the average value of the columns of the local displacement field presented in Fig. 6i. The horizontal axis is the position of each column, and the vertical axis is the average local displacement ${\varDelta {{\varvec{u}}}_{tm}}' \big (= \sum _{ i=1 }^{ N }{ {{\varDelta {{\varvec{u}}}_{t}}}' / N }\big )$ of each column, where N is the number of column of analysis area in Fig. 6i. The blue open circle shows the average error displacement caused by the error of the image registration ${\varDelta {{\varvec{u}}}_{tme}}'$ in the first column where it is expected the heated air does not flow. In this case, the location of the smartphone and the gas burner in Fig. 6 is similar to that in Fig. 5, where the position of the first column in Fig. 6 corresponds to the area of small displacement in Fig. 6. Therefore, the influence of the flame is also considered to be small. The detail of the error estimation due to the image registration is described in Appendix. The red open circle corresponds to the maximum average local displacement $\underline{{\varDelta {{\varvec{u}}}_{tm}}'} (= \max \{{\varDelta {{\varvec{u}}}_{tm}}'(m) | m = 1,2 \ldots M\})$ where the heated air is expected to flow. m means the natural number. The local displacement is large at the top of the flame. Figure 6iii presents the average error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ and the maximum average local displacement $\underline{{\varDelta {{\varvec{u}}}_{tm}}'}$ as a function of the smartphone moving distance. Three points in each z-position in Fig. 6iii mean three trials in the same condition. The horizontal axis shows the moving distance of the smartphone between the reference image and the target image. The depth movement of the smartphone [($-5, 5$) mm on the distance $Z_\mathrm{B}$] is defined negative when the smartphone approaches the gas burner and positive otherwise. It is assumed that a person holding a smartphone walks at 0.75 to 2 m/s (average human walking speed) (Donelan et al. 2002) and records the density gradient at 240 fps. Thus, the movement of the smartphone during one frame to the next frame is 3.1–8.3 mm. Therefore, it is thought that the depth movement of ±5 mm in Fig. 6iii is reasonable. The average local displacement is 1–2 pixels. Since the density gradient distribution caused by the flame changes due to turbulence of the air, the standard deviation of the local displacement increases. The ratio $Z_\mathrm{D}/Z_{B}$ in Eq. 1 increases when the smartphone approaches the gas burner. Therefore, it is considered that the maximum average local displacement $\underline{{\varDelta {{\varvec{u}}}_{tm}}'}$ is the largest when the depth movement of the smartphone is $-5$ mm in Fig. 6iii. The average error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ increases in proportion to the movement distance in the depth direction. The dynamic range ($\underline{{\varDelta {{\varvec{u}}}_{tm}}'}/{\varDelta {{\varvec{u}}}_{tme}}'$) in Fig. 6iii ranges $2<\underline{{\varDelta {{\varvec{u}}}_{tm}}'}/{\varDelta {{\varvec{u}}}_{tme}}' < 8$. This result indicates that SBOS can visualize the density gradient in the dynamic situation.

4.2.2 Visualizing the steady-phase object with the horizontal movement of a smartphone

It is expected that the density gradient of the steady-phase object can be visualized by moving the smartphone using SBOS. In this subsection, the steady-phase object is visualized with the horizontal movement of a smartphone. A glass plate as a steady-phase object is fixed on the floor. The smartphone is moved 5 mm horizontally to the right between the two images. Captured original images are shown in Fig. 7i and ii. The horizontal local displacement is shown in Fig. 7iii. The direction indicated by the x-axis arrow in Fig. 7iii is defined as a positive local displacement. The error displacement due to the image registration is observed in a whole area. However, in the area surrounded by the black dotted line in Fig. 7iii, the local displacements in the area of the glass plate are significantly different from the displacements in the area out of the glass plate. This is because the displacements in the glass plate area caused by the horizontal movement of the smartphone are not the same as those outside the glass plate area. This result suggests that SBOS could detect the density gradient of the steady-phase object.

Fig. 7

(i, ii) The original images. The smartphone moves 5mm horizontally while two images are captured. (iii) Local displacement in x component obtained from Fig. 7i and ii

Fig. 8

The time sequence of (i) the original image and (ii) the local displacement field (magnitude) obtained by SBOS using the embossed wallpaper as a natural background in the dynamic situation

Fig. 9

The time sequence of (i) the original image and (ii) the local displacement field (magnitude) obtained by SBOS in the dynamic situation. A color-patterned background plate is projected on the wallpaper

4.3 Visualization of density gradient without an mosaic-patterned background plate

Fig. 10

The time average of regularized local displacement field (magnitude). (i) SBOS with a background plate. (ii) SBOS without a background plate

Fig. 11

The time average of regularized local displacement on three doted lines in Fig. 10(i) and (ii). (i) Upper line. (ii) Center line. (iii) Bottom line. The blue line and the red line show the result with a background and that without a background plate, respectively

Fig. 12

The instant local displacement field without a background plate. (i) Example of detecting the area of density gradient. (ii) Example of not detecting the area of density gradient. The area surrounded by the dotted line is an enlarged view of the wallpaper

As a proof of concept of SBOS, the density gradient without an mosaic-patterned background plate is visualized in the dynamic situation. The wall pattern is utilized as a natural background (see Fig. 3ii). The local displacement field obtained by SBOS without the background plate and that obtained by SBOS using the background plate is compared in the dynamic situation.

The original image and the magnitude of the local displacement field obtained by SBOS using wallpaper are shown in Fig. 8i and ii, respectively. Here, a random-color background is not projected on the wallpaper. Since the flame of the gas burner is outside the screen of Fig. 8i, the position of the flame is indicated by an artwork. The smartphone is moving from left to right while the position of the gas burner is fixed. The spatial resolution is 2.8 pixels/mm and frame speed is 120 fps in the slow motion mode. The average movement velocity of the smartphone is about 0.22 m/s. The reference image (image C, Fig. 2) and the target image (image D, Fig. 2) are captured with time difference $\varDelta t$ (= 8.3 ms). The reference image is sequentially updated. In Fig. 9i, ii shows the time sequence of the original image and the magnitude of the local displacement field where the projected background plate is used. The experimental setup is the same as Fig. 8 except the background plate is projected. The spatial resolution is 2.8 pixels/mm and frame speed is 120 fps in the slow motion mode. The average moving speed of the smartphone is 0.21 m/s. Regardless of SBOS without the background plate or SBOS with the background plate, the time difference of the local displacement is clearly visualized.

The time average of local displacement fields of the heated air is shown in Fig. 10. For the time averaging, 48 sequential images (0.4 s) of the displacement fields with a background and 42 sequential images (0.35 s) of the displacement fields without a background are used. Since the position of the gas burner is different in all images, the local displacement field is aligned at the center of the image based on the position of the gas burner. For comparison, the displacement magnitudes are normalized by the maximum displacement of each averaged images, 0.8 pixel for a background case and 1.0 pixel for a no background case. In qualitative point of view, Fig. 10i and ii shows larger displacement areas near the center, where the flame is present. In contrast, lower displacement areas are at the sides of the averaged images; although for no background case, the large displacements are shown at the edge of the averaged image. This qualitative agreement indicates that SBOS can detect the area of the flame. For quantitative comparison, the normalized time average of local displacements at three different levels (denoted as doted lines in Fig. 10) is shown in Fig. 11. Although overall trend near the center (flame area) is similar as indicated by Fig. 10, the values near the edges of the images show significant difference. This is due to the non-trivial noise for the no background case, which was not averaged out since the number of images including this area is small (less than 5 images).

For further consideration, two instantaneous local displacement fields obtained by SBOS without background are shown in Fig. 12. The flame is located at the right corner in both images. In Fig. 12i, the local displacement is large only in the region above the flame, and the displacement is small in the other region. In this case, the wallpaper texture is clearly captured as shown in the enlarged view of Fig. 12i. In contrast, the local displacement is found in almost all areas of Fig. 12ii, including the area where the influence of the flame is expected to be small. In this case, the wallpaper texture is blurred as shown in the enlarged view. This blurred background image of the misty texture image could cause inaccurate cross correlations, resulting in large noise in the wallpaper case.

5 Conclusion and outlook

The smartphone BOS (SBOS) technique, which offers the measurement of the density gradient in the air using the high-speed imaging feature of the smartphone being carried with a moving observer, is proposed. SBOS cam obtain the time difference of the local displacement field by comparing a reference image and a target image captured at different times and position. The reference image is sequentially updated. In our experiment, the density gradient of heated air is visualized in both the static situation and the dynamic situation. When the smartphone moves in the depth direction, the ratio of the maximum average local displacement $\underline{{\varDelta {{\varvec{u}}}_{tm}}'}$ to the average error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ (i.e., dynamic range) is $2<\underline{{\varDelta {{\varvec{u}}}_{tm}}'}/{\varDelta {{\varvec{u}}}_{tme}}'< 8$. This result suggests that SBOS can visualize the time difference of the density gradient distribution in the dynamic situation. Furthermore, as a proof of concept of SBOS, the local displacement field obtained by SBOS with a natural background is estimated in the dynamic situation. Using an embossed wallpaper in a room as the natural background, SBOS clearly visualized the time difference of local displacement induced by a gas burner. Therefore, it shows that the SBOS is a promising mobile tool that can visualize the density gradient of the fluid.

SBOS can casually visualize the density gradient using only a smartphone and a reasonable background. As a possible future application, SBOS could be used for the education and measurement in labs with harsh environments, available to any smartphone users. It would be possible to execute all processes from the image capturing to the display of the result using only the smartphone in real time if SBOS computation gets fast enough.

Appendices

Appendix

Error estimation of image registration

Fig. 13

(i) The definition of coordinates corresponding to the movement of the smartphone. (ii) The local error displacement field obtained from the analysis area in moving situation without the density gradient

Fig. 14

The averaged error displacement obtained by the analysis area in Fig. 13 versus the horizontal, depth and roll movement of the smartphone. The horizontal axis on the lower side shows the parallel movement of the smartphone in the horizontal and the depth direction. The horizontal axis on the upper side shows the rotation angle of the smartphone. The black dotted line and the red dotted line show the averaged walking speed in the roll movement and the parallel movement, respectively

Fig. 15

A 3-D scatter diagram of the averaged error displacement in the situation where the camera moves by multiple axis. This figure includes plots obtained by the direction of horizontal depth, horizontal roll and depth roll, respectively

In this section, the accuracy of image registration with the average walking velocity of the human is estimated. When an observer captures snapshots with the smartphone at hand, the smartphone moves from the original position due to the influence of walking or shaking camera. The human walking speed $V_ \mathrm{h}$ is 0.75–2 m/s (Donelan et al. 2002). On the other hand, the acceleration due to camera shake [$= (V_{c2}-V_{c1})/(t_{2}-t_{1})$] during walking is $20\mathrm{m}/\mathrm{s}^{2}$ (García-Magariño et al. 2016). Here, $V_{c2}$ and $V_{c1}$ are the instantaneous movement velocity of the smartphone at time $t_{2}$ and time $t_{1}$. If the iPhone 7 (Apple Inc.) is used as a camera, 120 fps or 240 fps is selected. When the frame speed is 120 fps, and the acceleration of camera shake is 20 m/s$^{2}$, the difference of velocity $V_{c2}$–$V_{c1}$ is 0.16 m/s. The moving velocity of the camera plus the walking velocity $V_{h}$ and the camera shaking velocity $V_{c}$ is 2 ± 0.16 m/s. To verify the registration, an image used as a background plate without the density gradient is captured. The definition of moving direction is shown in Fig. 13i. The moving direction is the horizontal direction, the depth direction and the rotational direction with the center of image as rotation axis. The moving image is generated by translating, scaling, and rotating the original image photographed at the origin position on the software. Figure 13ii shows the result of registering the image captured at the origin position and the image captured after the stage movement and then computing the magnitude of local displacement by cross correlation. It is supposed that the magnitude of local displacement in Fig. 13ii is the error displacement ${\varDelta {{\varvec{u}}}_{te}'}$ since the local displacement is obtained by only an error of the registration. The plot in Fig. 14 shows the magnitude of average error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ = $\sum _{ j=1 }^{ M } \sum _{ i=1 }^{ N }{ {\varDelta {{\varvec{u}}}_{te}}'(i,j) / (M \times N) }$ in the analysis area (100 $\times$ 100 pixel$^{2}$) of the original image Fig. 13, where M and N are number of row and column, respectively. If the image positions before and after registration match perfectly, the average error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ is 0. The vertical dotted line indicates the movement of the smartphone corresponding to the average walking velocity in each movement direction. The movement velocity of the smartphone in the horizontal direction and the depth direction is obtained by $V_\mathrm{h}$ = lT, where l is the parallel movement of the smartphone and the frame rate is set at T = 120 fps.

The moving velocity of the smartphone in the rotation direction is described as $V_\mathrm{h}$ = $r\omega$ = $r\theta T$. Here, r is the distance from the center of the smartphone to the lens, $\omega$ is the angular velocity, and $\theta$ is the rotation angle of the camera. In Fig. 14, the average error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ in each direction is smaller than 0.22 pixels. The average error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ is large in the rotation direction and the depth direction. When an image is rotated or scaled around the image center axis, the average error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ of a certain point on the image is proportional to the distance from the center of the image. A large moving distance causes insufficient number of repetitions of registration, which is considered to degrade the accuracy of registration.

In the practical condition, the smartphone will move to multiple axes while capturing images. Figure 15 shows the average error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ when the smartphone moves to multiple axes. Each axis indicates the movement distance or the rotation angle of the smartphone with reference to the origin position. The moving distance 20 mm and the angle $8^\circ$ correspond to the average walking velocity of human. All average error displacement are within 0.2 pixels. The direction of the largest average error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ is a combination of the depth direction and the rotation direction. The average error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ in the rotation direction and the depth direction is large in the single axis movement. Therefore, it is considered that the average error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ increases in combination of two axes. In static condition, the error displacement ${\varDelta {{\varvec{u}}}_{tme}}'$ in area without the density gradient obtained by conventional BOS is less than 0.2 pixel (Richard and Raffel 2001), which is equivalent to our results. This indicates that the accuracy of image registration is sufficient for SBOS.

Effect of lens distortion

Fig. 16

A checkerboard used to analyze the coefficients of lens distortion

Fig. 17

(i) Undistorted image. (ii) Distorted image. (iii) The magnitude and the vector of local displacement by the distortion

The monochrome aberration is given such as the defocus, the spherical aberration, the coma, the astigmatism, the field curvature and the image (radial or tangental) distortion. The radial distortion is expected to be the main factor of aberrations because the iPhone is used the wide-angle lens. The edge of image is expanded or shrunken by the image distortion. Here, the additional measurement is conducted to estimate the radial distortion. First, the coefficient of lens distortion related to the radial distortion is estimated. Second, the displacement by comparing an image with the radial distortion and an image without the radial distortion is measured.

The radial distortion is modeled using the coefficient of lens distortion by the following equations (Bukhari and Dailey 2013):

$$\begin{aligned} { x }_\mathrm{{distorted} }=x ( 1+{ k }_{ 1 }{ r }^{ 2 }+{ k }_{ 2 }{ r }^{ 4 }+{ k }_{ 3 }{ r }^{ 6 } ) \end{aligned}$$

(6)

$$\begin{aligned} { y }_\mathrm{{ distorted }}=y ( 1+{ k }_{ 1 }{ r }^{ 2 }+{ k }_{ 2 }{ r }^{ 4 }+{ k }_{ 3 }{ r }^{ 6 } ) \end{aligned}$$

(7)

where ($x_\mathrm{{distorted}}$, $y_\mathrm{{distorted}}$) are the distorted pixel location and (x, y) are the undistorted pixel location. $k_{1}$, $k_{2}$ and $k_{3}$ are the coefficient of lens distortion. r is the distance from the center of the image. In our case, $k_{3}$ is not used in this estimation because $k_{1}$ and $k_{2}$ are sufficient for calibration. These coefficient can be calibrated by the experiment and the image analysis. The gCamera Calibrator h in Matlab is used for the calibration tool. This tool analyzes images capturing a checkerboard and detect checker patterns. Detected checker patterns are shown in Fig. 16. Red circles mean checker points. When the lens with the radial distortion is used, locations of checker patterns are slightly displaced from locations estimated geometrically. The coefficient of lens distortion is calibrated by detect this displacement.

In experiment, a $20 \,\mathrm{mm} \times 20 \,\mathrm{mm}$ checkerboard is used. Twenty images of a checkerboard placed at different locations are captured. The smartphone is at fixed position and autofocus is not used.

As a result, coefficients of ($k_{1}$, $k_{2}$) are (0.0637, $-\,0.1424$). The undistorted image is generated using these coefficients. An undistorted image and an distorted image (raw image of the patterned wallpaper) are displayed in Fig. 17i and ii, respectively. Note that Fig. 17i is generated from Fig. 17ii. Now, the local displacement field is calculated using both images using the cross-correlation method. The magnitude of local displacement field is shown in Fig. 17iii. The direction of displacement vectors is radially outward from the center of an image. This means that this radial distortion is the barrel distortion. The local displacement is small on the center region of an image and large on the edge region of an image. This result suggests that SBOS without the correction of the lens distortion is not suitable for the quantitative estimation. If SBOS for the quantitative estimation is required, the radial distortion should be corrected before the cross-correlation method in the image analysis.

Comparison between a smartphone and scientific camera

Fig. 18

Comparison of local displacement fields obtained by the smartphone and the scientific camera. (ii)–(iv) The magnitude of local displacement on horizontal lines in (i)

To compare the local displacements obtained with a smartphone (240 fps, $1280 \times 720$ pixels) and those with a scientific camera (FASTCAM SA-Z (Color), Photron Ltd., 250 fps, 1024 1024 pixels), both cameras are placed at the similar location and capture images simultaneously. The local displacement vector field of the air heated by a lighter is shown in Fig. 18i. The orange vector and the blue vector mean the local displacement obtained by the smartphone and the scientific camera, respectively. Both distributions of local displacement fields fairly agree. Figure 18ii–iv shows the magnitude of displacements obtained by the two cameras on the horizontal lines indicated in Fig. 18i, also showing reasonable agreement.