A comparative study of optical flow methods for fluid mechanics

Abstract

We present a benchmark study of Optical Flow (OpF) methods for fluid mechanics applications. It is aimed at assessing the performance of three OpF methods, Lucas and Kanade (in: Proceedings of the 7th 1519 international joint conference on artificial intelligence, 1981), Horn and Schunck (AI 17:185–203, 1981) and Farnebäck (Two-frame motion estimation based on polynomial expansion, in: Bigun, Gustavsson (eds) Image analysis, Springer, 2003), combined or not with the Liu and Shen (JFM 614:253–291, 2008) algorithm. The performance of the OpF methods, evaluated exclusively as the difference between the values of the methods and of ground-truth (reference values), is benchmarked for deformation dominated, rotation-dominated and uniform flows. For each flow type, relative and absolute errors are computed for different tracer displacements, noise levels, pixel particle sizes, image bit-depths and particle concentrations. The accuracy of the OpF methods seems mainly affected by the magnitude of velocity gradients and convective accelerations. It does not seem to be affected by the relative preponderance of rotational and deformation components. The inner region of the Poiseuille flow and the saddle point in the Rankine vortex combined with uniform flow pose significant difficulties to all methods. The performance of the Lucas-Kanade/Liu-Shen combination is the best for all flow types, image conditions and image bit depths. The Farnebäck/Liu-Shen combination has a similar high performance but only for image depths of 10 bit or higher. These OpF methods maintain a high performance for tracer sizes and concentrations outside the PIV optimal range. Horn-Schunck is the worst performing method, due to high sensitiveness to particle concentration variations or particle sizes. These results can be used to plan new particle-based velocimetry experiments or to retrieve further information from existing PIV databases.

Graphical abstract

Appendix

1.1 PIV and optical flow methods

This sub-section is dedicated to bring PIV, in a loose sense, into context with OpF. We present in Fig. 12 a brief comparison between JPIV v21.08 (Java), OpenPIV v0.23.6 (Python) and QuickLab PIV v0.5.7 (our reference implementation). From that figure we can see that all three PIV implementations perform better for particle concentrations of $N_i \ge 6$ (higher reliability) as well as with particle spot sizes of 2.0 or 3.0 px diameters (higher precision) except OpenPIV which also deals well with 6.0 px partice spot sizes. Furthermore, none of the implementations is able to cope well with 1.0 px particle spot sizes. These PIV implementations match the theoretical expected behavior for PIV methods (see Raffel et al. 2007).

Fig. 12

JPIV, OpenPIV and QuickLab PIV accuracy vs particle spot sizes and concentration ($N_i$). Results shown are obtained by averaging the SIGs relative error across all flow types at 4.0 px maximum displacement. Zero noise data is excluded since it is not realistic

OpF methods are compared with PIV by dividing all images in into interrogation areas of 16 px by 16 px, as employed by PIV software, rather than considering all the individual OpF vectors. Thus, a SIG absolute error and a SIG relative error are computed per each interrogation area based on the optical flow vectors that are contained in each interrogation area. Only at the final step, the average of the errors for all the interrogation areas is taken mimicking the error computation process that is made for the PIV accuracy evaluation.

Fig. 13

QuickLab PIV (PIV) versus dense Lucas-Kanade/Liu-Shen combined Pyramidal L2 box plots by flow type (8-bits) for the absolute ($\epsilon_{SIG}$) and relative ($\epsilon_{R-SIG}$) SIG errors

Observing Fig. 13 we see that the combined dense Lucas-Kanade/Liu-Shen performs well when compared to our PIV reference implementation, both in terms of SIG absolute and relative error. A few combinations of parameters work marginally better with our PIV reference implementation, particularly with the uniform flow and displacements of 4.0 px ($\mathscr {E}=-0.16 dBpx$ and $\mathscr {E}_R=-0.16 dB$). Nevertheless, those exceptions do not present relevant expression and it also does not mean that the tested OpF method is superior to existing state of the art PIV solutions. Our PIV implementation is a classic PIV implementation and will likely not perform on par with the current state of the art solutions.

Fig. 14

QuickLab PIV (PIV) versus Farnebäck/Liu-Shen combined Pyramidal L2 box plots by flow type (10-bits) for the absolute ($\epsilon_{SIG}$) and relative ($\epsilon_{R-SIG}$) SIG errors

The comparison chart in Fig. 13 confirms that the combined dense Lucas-Kanade/Liu-Shen method is valid for PIV processing of images with 8-bit depths. With an image bit depth of 10-bits it becomes possible to employ Farnebäck/Liu-Shen combination. The comparison results shown in Fig. 14 confirm that this method also performs well in comparison to our PIV reference implementation, both in terms of SIG absolute and relative error. The only relevant exceptions, albeit only in occurrence number, not in error value, occur for the uniform flow and displacements of 4.0 px where $\mathscr {E}=-0.28 dBpx$ and $\mathscr {E}_R=-0.28 dB$. From these results one can conclude that the Farnebäck/Liu-Shen combination is suitable for processing images with 10- or 12-bit depths. Our implementation of PIV does not show accuracy improvements with increasing bit depths above 8-bits.

1.2 Pyramidal versus non-pyramidal optical flow and bit-depths

Table 5 details the effects of employing pyramidal processing with the base algorithms at the various image bit depth levels, as a function of the maximum velocity considered (0.05 IA, 0.10 IA, 0.25 IA). It forms the base for the discussion of the pyramidal effects on accuracy. The values obtained for this table are averages of the SIG errors (both absolute and relative) as described in Sect. 3.

Table 5 Absolute and relative errors (Avg.) and standard deviations (Std.) of the base algorithms and of the base algorithms combined with 2-level pyramidal processing

From Table 5 it can be seen that the base version of Farnebäck is similar to its two level pyramidal version, while Horn-Schunck shows a small improvement for larger displacements of 4.0 pixels in respect to the two level pyramidal version. Instead, Lucas-Kanade has significant benefits with the two level pyramidal variant for the larger displacements. The distribution of the errors shown in Fig. 16 clarify the overall benefits of the pyramidal implementation. When comparing Lucas-Kanade against its own two level pyramidal version, we find that the pyramidal version is able to track 4.0 pixel displacements (0.25 IA width) much better than the non-pyramidal versions (see Table 5 and Fig. 15).

Table 6 discriminates the benefits across types of flow. It can be seen that dense Lucas-Kanade pyramidal version is especially better for the uniform flow, followed by the parabolic flow (see Table 6). Finally, rankine vortex, rankine with superimposed uniform and stagnation are only marginally improved. For smaller displacements, like 1.6 pixels (0.10 IA width) and 0.8 pixels (0.05 IA width), there is almost no measurable loss of accuracy, except a small error increment above the threshold for particles with 1.0 pixel diameter spots, with a particle number concentration per volume of 1 and a 15 dB WGIN (last entry of Table 6). Where, when considering a threshold of 1.0 dB for the $\mathcal{E}_R$ and 1.2 dB for the $\mathcal{E}$, these thresholds are slightly exceeded.

Fig. 15

dense Lucas-Kanade versus dense Lucas-Kanade Pyramidal L2 violin plots for the absolute ($\epsilon -SIG$) and relative ($\epsilon R-SIG$) SIG errors, all for 8-bit image depth

Table 6 Pyramidal L2 dense Lucas-Kanade improvements over dense Lucas-Kanade per flow type/displacement and bit depth. Absolute and relative error improvements are computed in accordance to Eq. 7.

Figure 15 shows that the pyramidal version provides accuracy improvements even at the lowest image bit-depth considered of 8-bits, where one can see that the errors are quite reduced. Moving to 10- or 12-bit image depths does not improve the accuracy for this method (see Table 6).

Table 7 Pyramidal L2 Farnebäck improvements over Farnebäck per flow type/displacement and bit depth. Absolute and relative error improvements are computed in accordance to Eq. 7.

Referring again to Table 5, but now regarding to Farnebäck versus two level pyramidal Farnebäck, it can be seen that the accuracy benefits only occur with a 4.0 px maximum displacement. The Farnebäck non-pyramidal method is already capable of tracking 4.0 px displacements, better than the other methods, just not as good as the pyramidal version.

From Table 7, the Pyramidal results for Farnebäck can be seen in more detail. The benefits occur specifically at 4.0 px maximum displacement for the uniform flow for all the bit-depths, and for the Poiseuille flow for 10 and 12-bit depths. This is the only case where 1.2 dBpx SIG absolute error improvement or the 1.0 dB SIG relative error improvement are exceeded, resulting in slightly improved accuracy. There are no accuracy drawbacks for any of the displacements, 0.8 px, 1.6 px or 4.0 px which exceed the considered thresholds ($> 1.2$ dBpx SIG absolute error or the $> 1.0$ dB SIG relative error improvements) for the tested bit depths. Figure 16 shows the accuracy improvement achieved for images with 12-bit image depths, being somewhat similar to what happens with 10-bit image depths, both with better results than the 8-bit image depths (see Table 7). For 8-bit image depths and 4.0 px displacements the gains are smaller, but the loss is not related with the pyramidal processing method, since the 8-bit depth non-pyramidal version also displays much worse performance than the higher bit-depths. As such, the accuracy loss lies in the base Farnebäck method, which does not perform so well for 8-bit depths.

Fig. 16

Farnebäck versus Farnebäck Pyramidal L2 violin plots for the absolute ($\epsilon_{SIG}$) and relative ($\epsilon_{R-SIG}$) SIG errors, all for 12-bit image depth

Fig. 17

Horn-Schunck versus Horn-Schunck Pyramidal L2 violin plots for the absolute ($\epsilon_{SIG}$) and relative ($\epsilon_{R-SIG}$) SIG errors, all for 8-bit image depth

Table 8 Pyramidal L2 Horn-Schunck improvements over Horn-Schunck per flow type/displacement and bit depth. Absolute and relative error improvements are computed in accordance to Eq. 7.

When considering Horn-Schunck versus two level pyramidal Horn-Schunck, and referring to Table 5, one can observe a reduction in the average errors accompanied with an increase in the standard deviation. Still, despite spreading the errors across a larger range, relevant accuracy improvements are obtained for 4.0 px displacements (see Fig. 17).

Table 8 further details the results for Horn-Schunck. From that table it can be seen that for 4.0 px maximum displacements the improvement occurs especially for the uniform flow, while also improving Poiseuille flow and slightly improving Rankine vortex, Rankine vortex with superimposed uniform flow and flow with stagnation point. The method has no accuracy reductions, for most displacements, except for one situation which comprehends a set of special conditions. It occurs for 8-bit depth and parabolic flow at 1.6 px maximum displacement and 3.0 px diameter particle at a particle volume concentration of 12 and 0 dB WGIN where a slight accuracy loss occurs exclusively for the SIG absolute error improvement that exceeds the 1.2 dB threshold. The SIG relative error improvement, for the same parameters combination, is still within the 1.0 dB threshold, meaning that the error is not significant when compared to the absolute vector magnitude (see Table 8). Moving to 10 or 12-bit image depths does not improve the accuracy for this method.

Horn-Schunck’s Lagrange multiplier parameter must be carefully selected for each pyramidal level, as it also significantly influences the attained results. In fact, it also significantly depends on the particle volume concentration and bit-depths.

Table 9 Absolute and relative errors (Avg.) and standard deviations (Std.) of the base algorithms combined with Liu-Shen without pyramidal processing versus with 2-level pyramidal processing

When combining Liu-Shen optical flow with each of the considered base algorithms we can verify the same trend regarding non-pyramidal versus 2-level pyramidal. Details can be found in table 9. When employing 2-level pyramidal for the Liu-Shen / Lucas-Kanade there are accuracy improvements similar to the ones observed for the base Lucas-Kanade as shown in Fig. 15. Regarding Liu-Shen / Farnebäck and its non-pyramidal versus 2-level pyramidal implementation, the behavior follows closely the one presented for the base Farnebäck like in Fig. 16. Finally, the results for the Liu-Shen / Horn-Schunck pyramidal and non-pyramidal version are once again, quite similar when comparing with the base Horn-Schunck algorithm in Fig. 17.