Improvement in systematic error in background-oriented schlieren results by using dynamic backgrounds

The use of electronic visual displays for background-oriented schlieren allows for the quick change of the reference images. In this study, we show that the quality of synthetic and background-oriented schlieren images can be improved by acquiring a set of images with different reference images and generating a median displacement field from it. To explore potential benefits, we studied different background changing strategies and their effect on the quality of the evaluation of the displacement field via artificial and experimental image distortions.


Introduction
Background-oriented schlieren (BOS) or synthetic schlieren imaging is widely used to measure density gradients in transparent media. In its most basic form, the only elements necessary for the experimental setup is a camera directed at a background. Changes to the density field between the camera and the background lead to different paths for the light and as a result to distortions of the captured image. One everyday example of this phenomenon is the image distortion caused by a hot surface. By comparing the distorted with an undistorted image of the background, the undistorted one called the reference image, the distortion and thus the density gradient can be numerically evaluated. Methods and experimental setups were first demonstrated by Dalziel et al. (1998Dalziel et al. ( , 2000 and Sutherland et al. (1999) under the name of synthetic schlieren and Raffel et al. (2000) as backgroundoriented schlieren. For the calculation of the displacement fields, dot tracking, pattern matching and optical flow algorithms were used. One of the key components of both synthetic schlieren and BOS is to find a suitable background for the object of study. This led to the development of a myriad of ways to find or generate backgrounds, ranging from natural backgrounds, such as open fields and trees, all the way to laser speckle backgrounds (Settles and Hargather 2017; Meier and Roesgen 2013) and multi-color backgrounds. In many of these cases, a more or less random pattern is used as a background. On the one hand, this randomness of a background is a key feature of the computational schlieren techniques, as it allows the differentiation of individual pixels. This is due to the random neighborhood of each pixel, which is used to identify the position of an image pattern in the distorted image relative to its position in the undistorted reference image with any of the aforementioned algorithms. On the other hand, this randomness is a potential source for systematic errors in the evaluation of the displacement field. If certain parts of the reference image in close proximity to each other display a similar or equal pattern around multiple pixels, a displacement in the distorted live image might be assigned to the wrong pixel by the algorithm. While this systematic error might be eclipsed in atmospheric conditions by statistical noise such as density fluctuations caused by air movement, the more care is taken to eliminate such influences, the more the systematic error due to the structure of the pattern has to be accounted for. Especially for the study of stationary distortions, such as the long-term measurement of nozzle flows or density gradients in optical elements, local inadequacies of the background pattern might be misinterpreted as the cause. Because of this dependency of a computational schlieren system on where on the reference image it is applied to, different approaches have been studied. Schroeder et al. used a digital mirror device to generate multiple random dot patterns (Schroeder et al. 2011).
Averaging the results of the individual reference images allows for the reduction of errors introduced by a singlereference image for a steady-state distortion of the density gradient field. Cozzi and Goettlich proposed the enhanced background-oriented schlieren (EBOS) technique in which several reference images are captured at different in-plane positions from the same background before the measurement is conducted (Cozzi and Goettlich 2019). During the measurement, the pattern matching algorithm is applied to compare the distorted image to the shifted reference images. By employing symmetric shifts, averaging the calculated displacement fields leads to an improvement in the mean error per pixel. Unfortunately, the approaches of Schroder and Cozzi increase the complexity of the measurement setup by implementing moving parts. A similar setup can also be used to improve and conduct reference free digital shadowgraphy (RFDS) (Gardner et al. 2020).
One way to more easily implement such strategies is the use of backlit monitors as backgrounds. Due to the decrease in size of individual pixels, improvements in contrast, brightness and resolution, as well as a decrease in price, the use of monitors and projectors for generating the background patterns have become viable. Through the use of backlit electronic visual displays, the setup of a BOS system can be simplified by eschewing the need for additional lighting of the background. The efficacy of backlit monitors was demonstrated by Sveen and Dalziel (2005) and more recently Wernet (2019). This use of digital backgrounds enables the user to display a wide variety of background patterns quickly and without mechanically changing the experimental setup, thus shortening the setup cycle and thereby increasing throughput. This was exploited by Spectabit Optics to develop a system which allows the automatic finding of the correct background algorithm combination (L'Esperance and Buckner 2017).
As proposed by Dalziel et al. (2007), we present in this study the effectiveness of using multiple different reference images to reduce the systematic error via numerical calculations and the use of a backlit monitor as a background to realize these improvements in an experiment. In both numerical study and experiment, the generation of the individual background patterns is based on 2D simplex noise and the displacement field is calculated via the optical flow algorithm by Farnebäck (2003), as implemented in the computer vision library OpenCV (Bradski 2000). The applicability of optical flow algorithms, albeit sparse optical flow algorithms, was studied and confirmed by Atcheson et al. (2009). For the numerical calculations, the distortion of the reference images is computer-generated, a technique established by Schreier et al. (2000), Atcheson et al. (2009) and more recently Rajendran et al. (2019). This allows for a study of the effectiveness and efficiency of different background changing schemes insulated from any random environmental effects. The presented schemes are based on varying the seed of the pattern generation function, changing the position of the background in the image plane or changing the resolution of the background pattern. In the experiments, the same background changing schemes are tested by measuring the displacement field introduce by placing a Fresnel lens between camera and background.

Methods
In this section, the algorithms used for the generation of the random patterns, as well as the kind of distortion introduced to the reference images, are presented. Furthermore, the algorithm employed to calculate the distortions from the images, and the calculation and evaluation of the discrepancies between the calculated and actual distortions are discussed.

Background generation
In this study, three types of patterns are used to check the potential of the proposed dynamic background scheme.
-A gradient pattern background, in which each pixel has an 8-bit grayscale value dependent on the grayscale value of its surrounding pixels. -A discrete pattern background, in which each pixel has a random 8-bit grayscale independent of its surrounding pixels -A binary pattern background, in which each pixel has randomly assigned a grayscale value of either 0 or 255.
The baseline pattern generation scheme used in this article is the 2D simplex noise, a gradient noise generation scheme. Developed by Ken Perlin, this scheme allows the generation of a computational inexpensive noise image without anisotropic artifacts (Perlin 2002). The simplex noise algorithm uses a triangular grid with a resolution coarser than the resolution of the reference image to be calculated and to be displayed in the BOS system. For the generation of a randomly patterned 2D image, each point of this grid is assigned two random values. Those values are used as gradient values along direction of the x-and y-axes between the points of the randomly patterned image. Starting from any point on the grid, a well-defined randomly patterned image with a continuous gradient can be achieved. For the implementation in our code the noise library in python is used, Fig. 1.
To create the individual gradient pattern reference images, this noise generation scheme is applied to a 512 px × 512 px grid. The parameters for the used implementation of the simplex noise are three equally weighted octaves with lacunarity steps of five with a unique randomization seed, Fig. 1a and d. The same algorithm is used to create the discrete pattern reference images. To achieve this, a triangular grid with a resolution the same or finer as the reference image is chosen. Thus, the calculation of each pixel is virtually independent from its neighboring pixels and leads to the creation of a random, discrete reference image, Fig. 1b and e. To create the binary pattern reference images, a threshold is applied to a discrete random pattern image. The grayscale value of each pixel is set to 0 if it is equal or smaller 127 and set to 255 if it is greater than 127, Fig. 1c and f. Using the same algorithm, reference images in which the grayscale values of neighboring pixels have a relation to each other (gradient pattern) and reference images with the grayscale values independent from pixel to pixel (discrete and binary pattern) can be created.

Artificial distortions applied to the background
In order to calculate the displacement field, there have to be distortions applied to the reference images. The displacement at an individual pixel position (i, j) is defined by the displacement vector with x and y denoting the displacement in the direction of the x-and y-axes. The distortions which are applied to the backgrounds are now described.

Discrete single-axis shift
This distortion of the reference image is used to evaluate the performance of the dynamic background scheme on extreme density gradients, such as can be found in nozzle flows. The discrete single-axis shift introduces discrete shifts to sections of the reference images. Each section, referred to as ribbon, has a width of 16 px perpendicular to the shift and spans the length of the reference in the direction of the shift. For the discrete single-axis shift distortion, 21 ribbons with shifts in the direction of the , y-axis and centered around x = 0 px are applied to the background. Each ribbon is shifted by one pixel relative to its neighboring ribbons, starting with y = −10 px at x = −168 px and ending with y = +10 px at x = +168 px, Fig. 2d. In the direction of the x-axis, no shifts occur, Fig. 2a.

Discrete double-axis shift
First, a discrete single-axis shift, identical to the one described above, is applied to the reference images. Therefore, the distortions in the direction of the y-axis, Fig. 2e, are identical to the ones applied via the discrete single-axis shift, Fig. 2d. On this shifted background an additional discrete single-axis shift, orthogonal to the preceding one, is performed. Because these shifts are applied to an already distorted reference image, the distortions in the direction of the x-axis are shifted as well, Fig. 2b.

Two-axis linear shift
This distortion of the reference images is used to evaluate the performance of the dynamic background scheme on a constant density gradient, which can be caused by optical lenses. The two-axis linear shift distortion is created by zooming into the middle of the image. In effect, the live image is a scaled up version of the respective reference image. The scaling up is achieved via a bilinear interpolation scheme, leading to a linear pixel shift parallel to each axis with a maximum pixel shift of x∕y = 10 px , Fig. 2c/f. In addition, the size of a pixel in the reference image is larger in the live image by a non-integer factor. Therefore, the grayscale values of the individual pixels in the reference are slightly smeared out and are bleeding into one another in the distorted image, as would be expected in a real live synthetic schlieren or BOS setup.

Optical flow
The Gunnar Farneback method is a dense optical flow algorithm (Farnebäck 2003;Bradski 2000). This means that the optical flow can be determined for each pixel. The algorithm is also very efficient and can be combined with multi-resolution options. This method uses polynomial extensions to detect the changes between two images. At each pixel of the background image, the values of its surrounding pixels can be approximately calculated using a quadratic polynomial with being a symmetric matrix, a vector and c a scalar. Applying this approach to the distorted image, a system of linear equations can be created, yielding an approximation of the pixel shift caused by the distortion. To improve the estimation of the pixel shift, multiple scales are used. Starting from a coarse scale, large pixel shifts can be estimated. These shifts are then used as a priori knowledge in the evaluation of the next smaller scale, thus refining the result with each step. In this work, the number of scale levels is six with a maximum averaging window size 64 px and a reduction in size by a factor of two for each scale level, down to singlepixel displacement.

Evaluation of the accuracy of the calculated displacement field
To quantitatively evaluate the benefits of the strategies, we use the average endpoint error ⟨e (k) ⟩ for each individual displacement field k, as introduced by Atcheson et al. (Atcheson et al. 2009). The value of ⟨e (k) ⟩ is calculated from e (k) ij , which is the magnitude of the difference between the estimated displacement vector (k) ij and the corresponding ground truth displacement vector true ij at each pixel with N x∕y the number of pixels along each axis. Similarly, the average endpoint error ⟨ẽ (n) ⟩ of the median of the estimated displacement vectors ̃ (n) ij over a set of n displacement fields is calculated Equivalently, the average endpoint error of the mean over a set of n displacement fields ⟨ē (n) ⟩ is calculated via The results of ⟨ẽ (n) ⟩ are used to compare quantitatively the results of the different background changing strategies, and between the median, mean and individual displacement field estimations. Additionally, we compare the displacement fields to the median displacement fields qualitatively and compare the effects on measurement artifacts.
For the relative improvements in the average endpoint, errors are compared to the initial value ⟨e (1) ⟩ . We use this metric, as it allows for a better comparison between the results of the median of the displacement field calculated from a single-or multiple-reference images. Even though this might not necessarily represent the worst, best, mean or median case, it best describes the results to be expected by someone switching from a static to a dynamic background approach and using the static reference image as the starting point and its results as the measurement stick.

Strategies for background changes
During standard synthetic schlieren or BOS measurements, only a fixed single background is used to capture both the reference image and the live images. Because the pattern of the background is generally random, this leads to areas where the reference image is well suited to resolve the distortions applied to it, while in other parts the pattern in the reference image is suboptimal for the resolution of the distortion. The errors incurred by this technique do not generally change and are therefore systematic. Hence, averaging over several images does not help for a static observation. (3) By choosing a reference optimally suited for the investigated case, this error can be reduced but not completely avoided, because of the random nature of the pattern. By changing and manipulating the reference images, the areas of nonoptimal calculation of the displacement, i.e., the areas where the systematic errors occur, are randomized. Thus, the systematic error is turned into a statistical error. As a result, the overall error can be reduced by creating the median or mean for each pixel of the displacement fields over several different reference images. Each strategy presented here is used for every type of reference pattern explored.

Position change
The first strategy presented is the lateral shift or position change of the reference image. In each iteration, the reference is shifted by 5 px in both the directions of the x-and y-axes. By creating the different reference images from a single static pattern, consecutive reference images are not completely independent from one another. While this correlation could be exploited, it also means that deviations from the ground truth, due to local insufficiencies of the reference pattern, are not independent between successive evaluations of reference/live image pairs. One advantage of this strategy is that it can be mechanically implemented as well and would free the image capture rate from the refresh rate of the digital display.

Pattern resolution
For this strategy, an initial pattern is created, which has a slightly larger size than the final reference images. The differing reference images are created from this initial pattern by increasing and decreasing the image size via a bilinear interpolation scheme and then cropped to the desired size. For the binary pattern references, a binarization step is applied after the scaling. Therefore, the resolution of the pattern is different for every reference image. While possible, we do not provide a weighting to the resulting shifts depending on the resolution of the noise.

Seed change
To increase the quality of the calculated displacement fields, the reference image for each iteration is created by using a different noise seed with the same background generation scheme, e.g., gradient/discrete/binary. Thus, for every iteration each pixel of the reference potentially takes on a different grayscale value and is surrounded by different grayscale values.

Strategy combinations
Finally, the strategies described above can be used in combination with each other. While the combined strategy might not be the optimal strategy, it increases the statistical nature of the incurred errors and might lead to a more robust improvement strategy.

Numerical evaluation of artificial distortions
To study the effectiveness of the background changing scheme without influence from the environment, the generated reference image and distorted image are compared directly. This removes any statistical effects due to camera noise or density gradients as a result of thermal shifts. Therefore, the resulting deviations from the ground truth in the individual estimates of the displacement field are only dependent on the reference image and the algorithm used for the evaluation of the distortions.

Discrete single-axis shift
For the discrete single-axis shift, the main contributors to ⟨e (k) ⟩ are the edges of the ribbons, with shifts in the direction of the y-axis, Fig. 3. Because the random patterns have a certain granularity, it leads to jagged edges for the individual evaluations of the displacement fields. This is especially pronounced in the areas with the extreme changes from | true y | = 10 px to true y = 0 px . This carries over to the estimation of the displacement in the direction of the x-axis. Because the shift cannot be correctly calculated for all points along the edges by the optical flow algorithm, artifacts appear in the calculated displacement fields.
By applying a multiple background strategy to resolve the image distortions, both the artifacts in the estimates for the displacements in the direction of the x-axis and y-axis can be suppressed, demonstratd here on binary pattern reference images with changing seeds, Fig. 3. The area at the transition between the shifts becomes more defined and the artifacts vanish.
This qualitative improvement is observable in the numerical estimates as well, Fig. 4a. While the values of ⟨e (k) ⟩ of the individual optical flow calculations are scattered around 0.25 px, the value of ⟨ẽ (n) ⟩ for the median displacement field is continuously getting smaller with each additional displacement field added to the calculation of the median. For 10 images, the value of ⟨ẽ (n) ⟩ is at 60% and converges to a value below 55% of its initial value ⟨e (1) ⟩ . Furthermore, the value of ⟨ẽ (n) ⟩ as a function of n is dominated by the deviations from the ground truth for displacements with a magnitude ‖ true ‖ ∈ (8, 10] and ‖ true ‖ ≤ 2 , Fig. 4b. As mentioned before, for ‖ true ‖ ∈ (8, 10] the high endpoint error is due to the transition between | true y | = 10 px to true y = 0 px . In the transition region, deviations from the ground truth, such as the observed jagged edges, incur larger errors compared to transitions between for example true y = 8 px and true y = 7 px . This transition contributes to ⟨ẽ (n) ⟩ for ‖ true ‖ ≤ 2 px as well but is weighted by the larger area with ‖ true ‖ = 0 px outside of the area with ribbons. For all other displacement values, the deviations from the ground truth decrease with an increase in the number of displacement fields used to calculate the median. The same behavior is observable for all types of patterns with this strategy, Fig. 4c and all strategies with the same type of pattern, Fig. 4d.
For the discrete single-axis shift, normalized by ⟨ẽ (1) ⟩ , the averaged value over all pattern and strategy combinations ⟨⟨ẽ (n) ⟩∕⟨ẽ (1) ⟩⟩ , relative to their respective initial values, is decreased to 68% for 10 images and 58% for 100 images, Fig. 5.   Fig. 4 Evaluation of the effects of background changing strategies on the mean deviations from the ground truth for the discrete single-axis shift distortion. a Average endpoint error for the individual ⟨e (k) ⟩ and median ⟨ẽ (n) ⟩ estimates of the displacement field, with a seed changing strategy on binary pattern references. b Value of ⟨ẽ (n) ⟩ , for the same strategy and type of pattern as a), calculated from pixels within a certain range of the magnitude of the true displacement field. c Value of ⟨ẽ (n) ⟩ for the seed changing strategy with different types of reference patterns. d Value of ⟨ẽ (n) ⟩ for a binary-type pattern reference with different background changing strategies

Discrete double-axis shift
The results for the evaluation of the discrete double-axis shift, are similar to those of the discrete single-axis shift, exemplified in this section with the results of a pattern resolution strategy with gradient pattern reference images. The displacement fields in the direction of the x-and the y-axes both display jagged edges at the transition between the ribbons. With an increase in the number of images used to calculate the median, these edges of the transitions become less jagged and more pronounced, Fig. 6a/b. Additionally, cross talk between the shifts along the two different axes is reduced and artifacts are suppressed. Because individual reference images are calculated from the same initial reference image, the values ⟨e (k) ⟩ are not randomly distributed but correlated, Fig. 7a. By changing the resolution of the background pattern, areas of the reference image, where the evaluation of the distortion works better or worse than average, are moved throughout the image changing process. Because the transitions between the ribbons are equidistant, this movement leads to a periodical change in the value of ⟨e (k) ⟩ , Fig. 7a. Independent of this periodicity, the value of ⟨ẽ (n) ⟩ decreases with the number of individual displacement fields used to calculate the median. Due to the additional pixel shifts in the direction of the x-axis, there are more transitions between areas with different magnitudes of displacement. Since deviations from the ground truth in the evaluation around these transitions incur a higher contribution to the average endpoint error, the absolute value of ⟨e (k) ⟩ and ⟨ẽ (n) ⟩ is increased compared to the discrete single-axis shift. This is mainly driven by the increase in transitions between areas with displacement magnitudes ‖ true ‖ ∈ (2, 8] , Fig. 7b, which occur in the overlap region of the shifts in the x-and y-axes. Because the overlap Fig. 5 Comparison of the relative improvement ⟨ẽ (n) ⟩∕⟨ẽ (1) ⟩ between the different strategies and types of patterns for the discrete single-axis shift. The average change relative to the initial value ⟨⟨ẽ (n) ⟩∕⟨ẽ (1) ⟩⟩ is marked with a dotted line, n = 10 , and a solid line, n = 100 region does not span the whole displacement field, there are still transitions between the ribbons with | true x,y | = 10 px and x,y = 0 px . Again, errors in the evaluation in the area around the transition lead to larger contributions to the average endpoint error compared to any other transitions. On the other hand, areas with ‖ true ‖ > 10 px only ever occur in the overlap region and are therefore surrounded at their sides by areas with similar magnitudes of displacement.
The smallest initial value ⟨e (1) ⟩ for the pattern resolution strategy was achieved with a binary-type reference image, with all reference images showing significant improvement relative to their respective initial value, Fig. 7c. The strategy which needed the fewest displacement fields Fig. 7 Evaluation of the effects of background changing strategies on the mean deviations from the ground truth for the discrete double-axis shift distortion. a Average endpoint error for the individual ⟨e (k) ⟩ and median ⟨ẽ (n) ⟩ estimates of the displacement field, with a resolution changing strategy on gradient pattern reference images. b Value of ⟨ẽ (n) ⟩ , for the same strategy and reference pattern as a), calculated from pixels within a certain range of the magnitude of the true displacement field. c Value of ⟨ẽ (n) ⟩ for the resolution changing strategy for different reference patterns. d Value of ⟨ẽ (n) ⟩ for gradient-type references with different background changing strategies (a) (b) (c) (d) Fig. 8 Comparison of the relative improvement ⟨ẽ (n) ⟩∕⟨ẽ (1) ⟩ between the different strategies and types of patterns for the discrete double-axis shift. The average change relative to the initial value ⟨⟨ẽ (n) ⟩∕⟨ẽ (1) ⟩⟩ is marked with a dotted line, n = 10 , and a solid line, n = 100 to converge onto its minimum ⟨ẽ (n) ⟩ is the pattern seed strategy, Fig. 7d. For the discrete double-axis shift, the averaged value over all pattern and strategy combinations ⟨⟨ẽ (n) ⟩∕⟨ẽ (1) ⟩⟩ , relative to their respective initial values, is at 62% of the initial value for 10 images and 51% for 100 images, Fig. 8.

Two-axis linear shift
Due to the linear nature of the distortion, there are no discontinuities in the true and calculated displacement fields, Fig. 9. Therefore, the deviations from the ground truth are on average relatively small compared to the discrete single-and double-axis shifts, Fig. 10. While ⟨ẽ (n) ⟩ , for all pattern and background changing strategies, shows a marked improvement, Fig. 11, and converges on the same value for all patterns and strategies, Fig. 10c/d, there are some strategies where ⟨ẽ (n) ⟩ is not monotone decreasing with the number of displacement fields used to calculate the median. This behavior can be observed for the resolution and position changing strategies. It is driven by areas in the reference image in which certain displacements are not correctly resolved. As one or multiple areas are moved through the reference background image by the resolution and position strategy, so does the resulting inaccuracy in the displacement fields. Because the distortions for the two-axis linear shift only slightly vary pixel to pixel, the problematic areas are only moved by small increments by the resolution and position strategy, the value of ⟨e (k) ⟩ is related to the ones before and after. Therefore, the rate at which the ⟨ẽ (n) ⟩ improves is smaller and not monotone, in case a suboptimal area is moved through the reference images. For this distortion and the presented position shift strategy, binary-type references yield the best results, and for the presented discrete-type pattern, the seed changing strategy is most successful in reducing ⟨ẽ (n) ⟩ . For the discrete two-axis linear shift, the averaged value over all backgrounds and strategy combinations ⟨⟨ẽ (n) ⟩∕⟨ẽ (1) ⟩⟩ , relative to their respective initial values, is at 60% of the initial value for 10 images and 45% for 100 images, Fig. 11.

Comparison between median and mean
As shown in the previous sections, using the median of the calculated displacement fields yields improvements across the board for all artificial distortions and background changing schemes. In general, the computational cost is higher for the calculation of the median than for the mean. In our calculations via the Python/NumPy implementation of these algorithms, the time to calculate the median is 5.4 times longer than for the calculation of the mean ( 1024 px × 1024 px displacement field calculation: 0.45 s , median: n = 5 ∶ 0.2 s, n = 10 ∶ 0.4 s, n = 20 ∶ 1.0 s , mean: n = 5 ∶ 0.02 s, n = 10 ∶ 0.04 s, n = 20 ∶ 0.08 s ). This is due to the calculation of the median being based on a sorting algorithm instead of simple summation for the calculation of the mean. As a result, all displacement fields have to be held in the memory to calculate the median, while for a comparable calculation of the mean the necessary memory could be reduced to the space needed for two images and two integer values. Therefore, a tangible benefit has to be achieved by the use of the median, compared to the mean, to justify the increased computational cost and memory usage. By applying a mean evaluation, the quality of the evaluation of the displacement field and the effectiveness of the multiple background strategies in reducing the value of ⟨e⟩ is decreased significantly, Fig. 12. This is due to the nature and influence of outliers on the estimation of the mean displacement field. As for the handling of deviating data points, in most displacement estimation schemes, and the one used here, the maximum calculable displacement is limited by the interrogation window size. If the true displacement is nonzero, this means that the maximum error of the estimation is smaller in the direction of the true displacement than in the opposite direction. Thus, if we assume a symmetrical distribution of the errors due to the background, the distribution of the errors of the estimate is skewed toward the opposite direction of the true displacement. This behavior is further increased if the algorithm for the evaluation of the displacement field returns a zero displacement if no proper displacement can be estimated. If a mean is applied to this skewed distribution, the calculated displacement is skewed Fig. 11 Comparison of the relative improvement ⟨ẽ (n) ⟩∕⟨ẽ (1) ⟩ between the different strategies and types of patterns for the two-axis linear shift. The average change relative to the initial value ⟨⟨ẽ (n) ⟩∕⟨ẽ (1) ⟩⟩ is marked with a dotted line, n = 10 , and a solid line, n = 100 toward zero as well. The deviation from the true displacement is greater, the smaller the difference between true displacement and the interrogation window is. In comparison, the evaluation of the displacement field via the median is more robust against the cutoff and miscalculation due to the interrogation window. As the median is the value at 50% of the data sample, the median does not display the same immediate skew toward zero as the mean for greater displacements from zero. Therefore, the average endpoint error for the evaluation via the median ⟨ẽ (n) ⟩ is smaller than for the evaluation via the mean ⟨ē (n) ⟩ , Fig. 12c. This effect is especially pronounced if only a small number of images is used, and each image has a larger weight. Only for a large number does this effect subside, thereby causing a slower decline in the average endpoint error.
The greater weight each individual evaluation of the displacement field has in the calculation of the mean displacement field leads to a smearing out of sharp gradients, Fig. 12a, compared to the displacement field calculated via the median, Fig. 12b. Therefore, if an averaging of the images is applied to a series of displacement fields calculated from the same image distortion, the use of the median yields better results than the use of the mean, especially with respect to the resolution of sharp edges.

Application to experiment
To demonstrate the effects of the use of background changing strategies in an experiment, they are applied to artificial distortions of the reference images and distortions created by a Fresnel lens. The artificial distortions allow for a quantitative evaluation of the effects of camera and display noise, as well as density fluctuations due to ambient conditions on the effectivity of the background changing scheme, as we will describe later. The evaluation of the displacement field of the Fresnel lens is used to demonstrate the effectiveness of this approach on a real-world application.

Experimental setup
To experimentally study the effects of the dynamic background scheme on the evaluation of the displacement field, a monitor-based background system is used. However, in such (c) Fig. 12 Evaluation of the effectiveness of the position changing strategy for the discrete double-axis shift on discrete-type reference images. a Displacement field in the direction of the y-axis, calculated via mean. b Displacement field in the direction of the y-axis, calculated via median. c Average endpoint errors with the resolution changing strategy for the individual displacement fields and the mean and median evaluation Fig. 13 Experimental setup using a laptop display for background generation, an USB camera for capturing and a Fresnel lens ( f = 350 mm ) to create an image distortion a setup the periodic display pixels in combination with the periodic camera pixels can lead to artifacts, known as Moiré patterns. To avoid Moiré patterns, the spatial resolution of the capturing system was limited by reducing the aperture size of the lens to a level at which the gaps between the pixels are not resolved and Moiré patterns are suppressed. Because this decreases the resolution of the displacement field as well, the displayed images were scaled up by a factor of 2 to counteract the impact of the loss of resolution. This allows for the investigation of the multiple background approach without having to take into account the effects of Moiré patterns on the evaluation.
To display the reference images, the monitor of a laptop, which is also used to conduct the optical flow calculations, is employed, Fig. 13. The monitor has a resolution of 1920 px × 1080 px at a diagonal length of 15.6 ′′ , which calculates to 141 ppi at a bit depth of 6 bit. The USB camera used to capture the images has a Sony IMX179 CMOS chip with a pixel unit cell size of 1.4 μm × 1.4 μm . The captured images have a bit depth of 8 bits in the MJPEG color format. In combination with a varifocal lens, the capturing system as a whole has a resolution of 600 lw mm −1 . In order to reduce possible optical aberrations, the captured image was cropped to an area 1024 px × 1024 px around its center. As we use the same reference images as for the strictly numerical evaluation of the artificial distortions, the reference images are scaled up by a factor of 2 with nearest neighbor interpolation.
The camera is positioned at a distance of 1100 mm and the lens is adjusted so that the displayed image is captured by an area of 1024 px × 1024 px which results in an effective focal length of 8.5 mm and a magnification of 0.0076. In this setup, the area of a single pixel in the numerical study has the size of 369 μm × 369 μm on the monitor and 2.8 μm × 2.8 μm.
To ensure no images were acquired during the changing of the reference image, both the display and the capture of images were performed consecutively in the same script with a timed delay between both steps.
While the degradation of contrast and resolution is always a concern for digitally captured images, the user of a system with a color display, a color camera with relatively small pixels and an optical system close to its resolution limit has to be careful not to lose too much image information in such a setup. In particular, the way color is presented and captured by digital devices lends itself to a loss of detail. In the presented case, this setup was chosen to demonstrate that the presented technique can be implemented and used with instruments that are at almost every laboratories disposal.
For the experiments with the Fresnel lens, the same setup is used, including the same reference images as before, with the Fresnel lens placed between the laptop display and the camera. The concentric groves of the Fresnel lens are spaced at an interval of 312 μm radially, with a focal length of f = 350 mm . The Fresnel lens is positioned so that it is in contact with the laptop display at the top and a distance of 20 mm at the bottom. To evaluate the effect on a displacement field with discrete gradient changes, the top right corner has been cut off. This is possible due to the thinness of the Fresnel which allows for this gradient step to be observable without any artifacts due to the transition, thus enabling the study of the effects of the multiple dynamic background approach on both continuous and discrete changes in the displacement field. In comparison, a flat plate of glass at an angle would only allow for the study of a discrete gradient change.

Measurement of artificial distortions
To quantitatively study the effectiveness of the presented approach, there is a need for knowledge of the ground truth of the displacement field. This is achieved here, by not only displaying the reference and the artificially distorted live images but also a grayscale image representing the artificial distortions. The grayscale value at each pixel matches the expected background shift at their respective position in the displayed artificially distorted live image, thereby allowing for the calculation of the deviation from the ground truth.
In order to map the grayscale values to the true displacement, the primary axis of the displayed images was aligned with the corresponding axis in the captured image during the experimental setup. Afterward the grayscale values for the discrete shifts are reduced to 21 levels in both axes, while the grayscale values for the two-axis linear shift are created via linear interpolation from scratch. Due to the realities of experimentation, there is a small difference in size between a displayed and captured pixel. Therefore, the discrete shifts do not occur as perfect integer values in the calculated displacement field. To account for this, an offset and a multiplication factor are applied to the captured ground truth displacement field.
Because the live images are captured with a camera from the laptop display, camera and display noise are introduced to the evaluation of the displacement field and the average endpoint errors. To compare the influence of the random deviations, due to noise introduced by the experimental setup and the systematic inadequacies of the chosen reference image, multiple distorted live images with a single-static-reference image and multiple distorted live images with multiple different reference images are captured. Applying the median calculation approach to multiple measurements of a single-reference image here with an artificial discrete double-axis shift with a discrete-type pattern shows little improvement. Even after calculating the median of 100 images, the edges of the ribbons are jagged and the artifacts from the image shifts in the x-direction are clearly discernible, Fig. 14b. While the value of ⟨ẽ (n) ⟩ generally decreases with an increase of images in the median calculation, Fig. 14c, the improvement is relatively small with a single-reference image in comparison with the multiple-reference approach. By using multiple different reference images, the borders of the ribbons can be identified and the artifacts from the x-axis shifts are suppressed, Fig. 14a. For the dynamic background approach, the value of ⟨ẽ (n) ⟩ converges on 48% of the initial value ⟨e (1) ⟩ , while using a single-reference image results in an improvement to 80% of the value of ⟨e (1) ⟩ . For this kind of artificial distortion, the improvement in the image quality and decrease in the average endpoint error are greater for the median of multiple-reference images than for the median of multiple images of a single-reference image.
Applying the same approach to a two-axis linear shift distortion yields a slightly different result. Qualitatively, both displacement fields display less noise and fewer artifacts from the shifts in the direction of the x-axis. Only in the corners of the displacement field, where the displacement is highest, does the multiple-reference approach outperform the singlereference approach, Fig. 15a/b. For displacements with small gradients both the single-reference and multiple-reference approach lead to an improvement in the estimate of the ground truth relative to the initial estimate ⟨e (1) ⟩ , 52%, Fig. 15c, with the multiple-reference approach performing generally better, 28%. The reason for the difference lies in the mechanism by which the displacement field is improved by the single-reference approach. By adding camera and electronic display noise to the capture of the live images, uncertainty is introduced to the estimate of the displacement field. If there are gradients inside the displacement field, this uncertainty leads to a smoothing of the difference between two neighboring pixels. In case of constant shifts or a linear increase in the shifts, this smoothing leads the estimate of the displacement field to become more accurate. Inversely, if there are steps in the true displacement field, then they are smoothed out by the singlereference approach as well, thereby decreasing the accuracy of the estimate. In conclusion, the improvements due to the single-reference approach are caused by the smoothing out of steps in the estimate of the displacement field. Therefore, the averaging with only a single-reference image yields significant improvements for distortions with only small differences in the displacement between neighboring pixels, such as the two-axis linear shift discussed here.

Measurement of Fresnel lens
Finally, the multiple-reference approach is applied to a realworld image distortion, caused by a Fresnel lens. In both cases, the reference images are of the binary-type pattern kind. For the multiple-reference approach, the seed changing strategy is employed. For a single displacement field, the borders of the Fresnel lens are not properly resolved and display the same jagged edges as for the discrete shifts and are generally noisy, Fig. 16a. With each displacement field added to the calculation of the median, the random fluctuations are decreased and 14 Comparison between the seed changing strategy and the single-reference image approach for the discrete double-axis shift applied to discrete pattern reference images. a Displacement field in the direction of the y-axis, calculated with the seed changing strategy. b Displacement field in the direction of the y-axis, calculated with a single-reference image approach. c Average endpoint errors of the seed changing strategy, the single-reference image approach and the individual displacement fields and the mean and median evaluation (a) (b) (c) Fig. 15 Comparison between the seed changing strategy and the single-reference approach for the two-axis linear shift applied to discrete pattern reference images. a Displacement field in the direction of the y-axis, calculated with the seed changing strategy. b Displacement field in the direction of the y-axis, calculated with a single-reference approach. c Average endpoint errors of the seed changing strategy, the single-reference approach and the individual displacement fields and the mean and median evaluation (a) (b) (c) (d) Fig. 16 Displacement field in the direction of the y-axis, created by the application of the seed changing strategy with binary pattern reference images. a Displacement field calculated from a single live image. b Median of 10 different reference images. c Median of 100 different reference images. d Zoom in of c the borders of the Fresnel lens become more easily discernible, Fig. 16b. At 100 images, only areas with a high pixel displacement are not correctly resolved, due to exceeding the interrogation window of the optical flow algorithm, Fig. 16c/d. In contrast, the application of the median to multiple images with the same reference image does not lead to any significant improvement to the evaluation of the displacement field. Again it is shown that the multiple backgrounds approach leads to an improvement in the estimate of the displacement field, compared to just applying the mean or median to multiple calculated displacement fields with only a single-reference image (Fig. 17).

Conclusion
In this article, we have shown that the use of dynamic background changing strategies in combination with the calculation of the median displacement field leads to an improvement in the average deviation from the ground truth ⟨ẽ (n) ⟩ and the quality of the estimate of the displacement fields. Via a strictly numerical approach, we were able to separate the evaluation of this strategy from any statistical influences. For all artificial distortions, the presented strategies yielded a significant improvement in the quality of the calculated displacement field. Artifacts from cross talk between the displacements in the different axis were successfully suppressed and transitions between areas with different magnitudes of displacements improved. Quantitatively, an improvement in ⟨⟨ẽ (n) ⟩∕⟨e (1) ⟩⟩ over all tested configurations of 64% with 10 images and 52% with 100 images was achieved. In general, the strategies with the most reliable improvement were the ones involving a seed changing strategy. For this strategy, each reference image is independent of the others. Therefore, the systematic errors due to insufficiencies of the reference images are completely transformed into statistical errors. Both the position and resolution changing strategies retain a dependence between individual reference images. As a result, the transformation from systematic to statistical errors is not complete. Additionally, we showed that the calculation of the displacement field via the median yields better results than the calculation via the mean, both qualitatively and quantitatively. This is due to the influence of strongly deviating data points on the calculation of the mean. Their influence leads to a smearing out in the mean calculated displacement field at positions with sharp gradients or large displacements compared to the calculation via the median.
By using a backlit monitor to display both the reference and the distorted live images, we were able to reproduce the same results semi-experimentally. Capturing the displayed images with a camera from the monitor introduced