1 Introduction

In fluid mechanics, singular value decomposition (SVD) and proper orthogonal decomposition (POD)Footnote 1 are used for particle image velocimetry (PIV) data regression (Sherry et al. 2013; Raiola et al. 2015; Mendez et al. 2017; Brindise and Vlachos 2017); identification of coherent structures (Kriegseis et al. 2009; Druault et al. 2012; Kourentis and Konstantinidis 2012; Marié et al. 2013; Xu et al. 2013; Gandhi et al. 2015); low-order modeling, possibly for flow control (Ma et al. 2003; Feng et al. 2011; Feng and Wang 2014); optimal sensor placement (Cohen et al. 2004); dynamic mode decomposition (Schmid 2010; Dawson et al. 2016); and more. In (Epps and Krivitzky 2019), we considered mode corruption due to noise present in the data, building on the works of Kato (1976), Breuer and Sirovich (1991), Venturi (2006) and Epps and Techet (2010). Herein, we consider use of the SVD/POD for data de-noising.

The SVD and POD are very attractive for de-noising experimental velocity field data, because no other rank r approximation captures more of the kinetic energyFootnote 2 in the data as the sum of the first r SVD modes (Schmidt 1907). However, when the data are corrupted by noise, it is a priori unclear how badly the SVD modes are corrupted by the noise and which modes to use when noise filtering.

Motivating questions for this article include the following: How effective is the SVD for filtering out noise and reconstructing the clean data? What method yields the most accurate reconstruction, and what are the limits to its accuracy? Can the magnitude of the noise be inferred from the noisy data themselves?

To define our goals mathematically, consider the following definitions: Let \(\mathbf{A}\in {\mathbb {R}}^{T\times D}\) be a matrix of clean data, with T time steps and D data sites (\(T < D\)). The reduced singular value decomposition of \(\mathbf{A}\) is

$$\begin{aligned} \mathbf{A} = \mathbf{U} \, \mathbf{S} \, \mathbf{V}^\intercal = \textstyle {\sum _{k=1}^T}\, \varvec{{\mathbf{u}}}_k s_k \varvec{{\mathbf{v}}}_k^\intercal ~ , \end{aligned}$$

where matrices \(\mathbf{U} \in {\mathbb {R}}^{T\times T}\) and \(\mathbf{V} \in {\mathbb {R}}^{D \times T}\) are each orthogonal and \(\mathbf{S} \in {\mathbb {R}}^{T\times T}\) is diagonal. The kth SVD mode consists of a left singular vector (temporal mode shape) \(\mathbf{u}_k \equiv U_{1{:}T,k}\), a singular value \(s_k \equiv S_{kk}\), and a right singular vector (spatial mode shape) \(\mathbf{v}_k \equiv V_{1{:}D,k}\). Let \(\varvec{\tilde{\mathbf{A}}}\in {\mathbb {R}}^{T\times D}\) be a matrix of noisy data:

$$\begin{aligned} \varvec{\tilde{\mathbf{A}}}= \mathbf{A}+ \mathbf{E} ~ , \end{aligned}$$

where \(\mathbf{E} \in {\mathbb {R}}^{T\times D}\) contains random noise with zero mean and standard deviation \(\epsilon\).Footnote 3 Let \(\varvec{\tilde{\mathbf{A}}}= \varvec{\tilde{\mathbf{U}}}\, \varvec{\tilde{\mathbf{S}}}\, \varvec{\tilde{\mathbf{V}}}^\intercal = \sum _{k=1}^T \varvec{\tilde{\mathbf{u}}}_k {\tilde{s}}_k \varvec{\tilde{\mathbf{v}}}_k^\intercal\) and \(\mathbf{E}= \varvec{\acute{\mathbf{U}}}\, \varvec{\acute{\mathbf{S}}}\, \varvec{\acute{\mathbf{V}}}^\intercal = \sum _{k=1}^T \acute{\varvec{{\mathbf{u}}}}_k \acute{s}_k\acute{\varvec{{\mathbf{v}}}}_k^\intercal\).

A reconstruction of the clean data can be formed by summing the first r SVD modes:

$$\begin{aligned} \varvec{\bar{\mathbf{A}}}_r \equiv \sum _{k=1}^r \varvec{\bar{\mathbf{u}}}_k \bar{s}_k \varvec{\bar{\mathbf{v}}}_k^\intercal ~ , \end{aligned}$$

where these barred variables could be the noisy (tilde) variables or any estimate of the clean ones. To evaluate such a reconstruction, we use the reconstruction loss

$$\begin{aligned} \varDelta _r \equiv \Vert \mathbf{A}- \varvec{\bar{\mathbf{A}}}_r \Vert _F^2 = \sum _{t=1}^T \sum _{d=1}^D (A_{td} - ({\bar{A}}_r)_{td})^2 ~ , \end{aligned}$$

where \(\Vert \cdot \Vert _F\) is the Frobenius norm, so (4) is a measure of the difference between the clean data and the reconstruction.

Herein, we consider being given noisy data \(\varvec{\tilde{\mathbf{A}}}\) but not knowing \(\mathbf{A}\) and \(\epsilon\). Our goals are (1) to make an accurate estimate \({\bar{\epsilon }}\) of the measurement error; and (2) to form a minimum-loss estimate \(\varvec{\bar{\mathbf{A}}}_r\) of the clean data.

1.1 Existing noise filtering methods

A number of methods exist to choose which SVD modes to use for noise filtering. Epps and Techet (2010) reconstruct an estimate of the clean data using all noisy SVD modes for which

$$\begin{aligned} {\tilde{s}}_k > \epsilon \sqrt{T D} ~ , \end{aligned}$$

where \({\tilde{s}}_k\) is the singular value of noisy mode k, \(\epsilon\) is the root mean square error in the data, T and D are the number of timesteps and data sites, respectively.

Raiola et al. (2015) use all noisy SVD modes for which

$$\begin{aligned} F_k \equiv \frac{s_k^2 + \epsilon ^2 D}{s_{k-1}^2 + \epsilon ^2 D} < 0.999 ~ . \end{aligned}$$

Note that (6) can not be used in practice, since one does not know the clean singular values (if only given a noisy dataset). However, (6) can be put into usable form by noting that Raiola et al. (2015) derived (6) assuming \({\tilde{s}}_k^2 \approx s_k^2 + \epsilon ^2 D\), so the spirit of (6) is equivalent to

$$\begin{aligned} {\tilde{F}}_k \equiv \frac{{\tilde{s}}_k^2}{{\tilde{s}}_{k-1}^2} < 0.999 ~ . \end{aligned}$$

Brindise and Vlachos (2017) do not sum noisy modes \(k = 1,\ldots ,r\) but instead choose the r “smoothest” modes, as identified by the Shannon entropy. Their so called entropy line fit (ELF) method is a six step procedure:

  1. 1.

    Compute the discrete cosine transform (DCT-II) of each right singular vector,

    $$\begin{aligned} {\hat{V}}_{dk} = \sqrt{\tfrac{2/N}{1+\delta _{d1}}} \sum _{i=1}^D V_{ik} \cos \left[ \frac{\pi }{D}\left( i-\tfrac{1}{2}\right) (d-1)\right] ~ , \end{aligned}$$

    for \(d = 1,\ldots ,D\) and \(k = 1,\ldots ,T\).

  2. 2.

    Square and normalize the DCT coefficients:

    $$\begin{aligned} X_{dk} = {\hat{V}}_{dk}^2 ~ /~ {\textstyle \sum _{d=1}^D} {\hat{V}}_{dk}^2 ~ . \end{aligned}$$
  3. 3.

    Compute the Shannon entropy of the normalized DCT signalFootnote 4

    $$\begin{aligned} Y_k = - \sum _{d=1}^D X_{dk} \log _2 X_{dk} ~ .\end{aligned}$$
  4. 4.

    Sort the minimum entropy into ascending order, \({\bar{Y}}_k\).

  5. 5.

    Use a “two-line fit” procedure to find the the optimal reconstruction rank, \(r_B\), where the sorted entropy \({\bar{Y}}_k\) nearly reaches a plateau.

  6. 6.

    Use sorted modes \(1,\ldots ,r_B\) for data reconstruction.

Note that while the Schmidt theorem guarantees that this reconstruction \(\varvec{\bar{\mathbf{A}}}_r\) will yield more loss to the noisy data \(\Vert \varvec{\tilde{\mathbf{A}}}- \varvec{\bar{\mathbf{A}}}_r \Vert _F^2\) than that using unsorted modes \(1,\ldots ,r\), this reconstruction might (or might not) yield less loss to the clean data \(\Vert \mathbf{A}- \varvec{\bar{\mathbf{A}}}_r \Vert _F^2\).

Shabalin and Nobel (2013) reconstruct the clean data using all the modes, but they discount the singular values in such a way as to suppress the more noisy modes.

Note that the above references (as well as this manuscript) assume a complete dataset with no large outliers. Other papers have considered POD for reconstruction of gappy data or data with extreme outliers (Venturi and Karniadakis 2004; Wang et al. 2015; Higham et al. 2016).

2 Proposed ‘E15’ noise filtering method

The present work builds upon (Epps and Krivitzky 2019), wherein we derived and validated a theoretical prediction (13) of the root mean square error of the SVD modes, which is defined as

$$\begin{aligned} \text {rmse}(\varvec{\tilde{\mathbf{v}}}_k) \equiv \left[ \frac{1}{D}\sum \limits _{i=1}^D \left( {\tilde{V}}_{ik} - {V}_{ik}\right) ^2 \right] ^{\frac{1}{2}} . \end{aligned}$$

Further, we showed that for modes constituting random noise, the \(\text {rmse}(\varvec{\tilde{\mathbf{v}}}_k)\) reaches a maximum value of \(\sqrt{2/D}\). The key idea of our proposed noise filtering method is to use only the modes for which the \(\text {rmse}(\varvec{\tilde{\mathbf{v}}}_k)\) is sufficiently below this noise ceiling.

Herein, we propose the following method for minimum-loss noise filtering.

  1. 1.

    Given a noisy data matrix \(\varvec{\tilde{\mathbf{A}}}\), perform the SVD (Matlab svd command) to obtain \(\varvec{\tilde{\mathbf{u}}}_k\), \({\tilde{s}}_k\), and \(\varvec{\tilde{\mathbf{v}}}_k\).

  2. 2.

    Estimate the measurement error \({\bar{\epsilon }}\) and the ‘spatial correlation parameter’ f by fitting a Marchenko–Pastur distribution to the tail of the noisy singular values. See “Appendix  1” for details.

  3. 3.

    (Optional) Infer the ‘effective smoothing window width’, w, from the curve fit in Fig. 14b, which reads

    $$\begin{aligned} w = 1 + \left( 2f - \tfrac{3}{2} \right) \left( 1-\mathrm {e}^{-20(f-1)}\right) ~ . \end{aligned}$$

    See “Appendix  1” for discussion of w. For PIV data, however, we recommend that (12) not be used and instead to set \(w=1\) (see PIV examples in Sects. 46).

  4. 4.

    Estimate the root mean square error of the modes:

    $$\begin{aligned} \begin{aligned} \text {rmse}(\varvec{\tilde{\mathbf{v}}}_k)\approx \text {min}&\Bigg \{ \sqrt{2/D} ~, \\& \frac{{\bar{\epsilon }}}{{\tilde{s}}_k} \, \Bigg [ \frac{D-w}{D} + \frac{w}{D} \mathop {\sum \limits _{m=1}}\limits _{m\ne k}^T \frac{ \tilde{\lambda }_m(3\tilde{\lambda }_k - \tilde{\lambda }_m) }{(\tilde{\lambda }_m - \tilde{\lambda }_k)^2} \Bigg ]^{\frac{1}{2}} \Bigg \} ~ , \end{aligned} \end{aligned}$$

    where \(\tilde{\lambda }_k \equiv {\tilde{s}}_k^2\). This formula was derived and validated in (Epps and Krivitzky 2019).

  5. 5.

    Estimate the rank for minimum-loss reconstruction as follows:

    $$\begin{aligned} t_k&\equiv \frac{ \log (\text {rmse}(\varvec{\tilde{\mathbf{v}}}_k)) - \log (\sqrt{2/D}) }{ \log (\text {rmse}(\varvec{\tilde{\mathbf{v}}}_1)) - \log (\sqrt{2/D}) } , \end{aligned}$$
    $$\begin{aligned} \boxed {{\bar{r}}_{\min } \equiv \text {maximum }k\text { such that }\,~ t_k > 5\% } ~ . \end{aligned}$$

    The parameter \(t_k\) in (14) quantifies the cleanliness of a mode, where \(t_1 = 1\) for the first (cleanest) mode, and \(t_k = 0\) for modes at the noise ceiling \((\text {rmse}(\varvec{\tilde{\mathbf{v}}}_k) = \sqrt{2/D})\). Modes that are sufficiently below this noise ceiling (i.e. that have a large enough \(t_k\)) are deemed clean enough to be useful for data reconstruction. The threshold in (15) was set at \(5\%\), because we empirically found that level to yield accurate and robust results.

  6. 6.

    Reconstruct an estimate of the clean singular values:

    $$\begin{aligned} \bar{s}_k = {\left\{ \begin{array}{ll} \sqrt{ {\tilde{s}}_k^2 - (\epsilon '{\hat{s}}_k)^2 } &{} \quad (k < k_c) \\ 0 &{} \quad (k_c \le k) \end{array}\right. } ~ , \end{aligned}$$

    where \(\epsilon '{\hat{s}}_k\) is a Marchenko–Pastur distribution (see “Appendix 1”), and

    \(k_c\) is the minimum index k such that \({{\tilde{s}}}_{k} < \epsilon '{\hat{s}}_k\) (Epps 2015). This cutoff index ensures that (16) yields a real number, and it sets \(\bar{s}_k\) to zero for modes in the tail of the distribution, which are obliterated by noise. Equation (16) follows from the observation that \({\tilde{s}}_k^2 \approx s_k^2 + (\epsilon '{\hat{s}}_k)^2\).

  7. 7.

    Reconstruct an estimate of the clean data via

    $$\begin{aligned} \varvec{\bar{\mathbf{A}}}_r = \sum _{k=1}^r \varvec{\tilde{\mathbf{u}}}_k \bar{s}_k \varvec{\tilde{\mathbf{v}}}_k^\intercal ~ , \end{aligned}$$

    with \(r = {\bar{r}}_{\min }\) from (15) and \(\bar{s}_k\) from (16). We will refer to (15)/(16)/(17) as the ‘E15’ reconstruction.

2.1 Illustrative example

To illustrate this procedure by way of example, consider a laminar, unsteady, 2D flow past a cylinder at Reynolds number \(\textit{Re} = V_\infty d/\nu = 100\). A clean (noise-free) dataset \(\mathbf{A}\) is extracted from CFD simulations, with \(D = 33{,}284\) data sites (x and y velocities at 16,642 grid points) and \(T = 455\) timesteps. The time-mean were not removed from these data. A noisy dataset \(\varvec{\tilde{\mathbf{A}}}\) is created by adding independent, identically-distributed (i.i.d.) Gaussian noise with standard deviation \(\epsilon = 10^{-4}\). Since the noise is i.i.d., \(w=f=1\).

Figure 1 provides a snapshot in time of the velocity magnitude, showing the clean data, noisy data, and a data set formed by the ‘E15’ reconstruction procedure (15)/(16)/(17). This reconstruction filters out much of the noise and faithfully represents the clean data.

Figure 2a illustrates the singular values of the clean and noisy datasets, as well as the ‘E15’ reconstruction (16) and a best-fit Marchenko–Pastur distribution. Equation (31) uses this best-fit Marchenko–Pastur distribution to predict \({\bar{\epsilon }}= \epsilon ' = 1.0019 \times 10^{-4}\), just 0.19 percent greater than the true \(\epsilon\).

Figure 2b shows the root mean square error of the spatial mode shapes. The theoretical prediction (13) agrees well with the numerically-computed rmse. The rmse is very low for the first mode and reaches \(\sqrt{2/D}\) for modes that are saturated with noise. The optimum reconstruction rank \({\bar{r}}_{\min }=9\) (15) is illustrated as the index for which the rmse approaches 5% of this noise ceiling, since modes with higher rmse constitute noise.

Fig. 1
figure 1

(Sect. 2.1 example) Snapshot of velocity magnitude, showing the clean data (top), noisy data (middle), and the ‘E15’ reconstruction (bottom). Flow is left to right

Fig. 2
figure 2

(Sect. 2.1 example) a Singular values. b Root mean square error, comparing theory (13) versus numerical results, and illustrating the optimum rank prediction \({\bar{r}}_{\min }\) (15). c Loss (4) for reconstructions using the summands shown in the legend

Figure 2c presents the loss \(\varDelta _r\) (4) versus rank r for several reconstruction methods. For example, using the noisy singular values and vectors, \(\varvec{\bar{\mathbf{A}}}_r = \sum _{\ell =1}^r \varvec{\tilde{\mathbf{u}}}_\ell {\tilde{s}}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal ,\) is labeled ‘\(\varvec{\tilde{\mathbf{u}}}_\ell {\tilde{s}}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\)’. As expected, the loss for the ‘\(\varvec{\tilde{\mathbf{u}}}_\ell {\tilde{s}}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\)’ reconstruction (red curve) has a pronounced minimum, because the lower modes contain most of the signal, and the higher modes contain mostly noise.

The proposed reconstruction method (15)–(17) is labeled ‘\(\varvec{\tilde{\mathbf{u}}}_\ell \bar{s}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\) E15’ in Fig. 2c. Figure 2c shows the loss for the ‘E15’ reconstructions at all ranks (green dashed curve), but note that the ‘E15’ method selects rank \({\bar{r}}_{\min }=9\) (per 15). There are two key points to emphasize: First, the ‘E15’ loss at \({\bar{r}}_{\min }\) is slightly lower than the minimum loss achievable using the standard ‘\(\varvec{\tilde{\mathbf{u}}}_\ell {\tilde{s}}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\)’ reconstruction (red curve). Moreover, the ‘E15’ loss at \({\bar{r}}_{\min }\) is only 1.7% higher than the minimum ‘E15’ loss (which actually occurs at \(r_{\min }=11\)).

Second, the main advantage of the ‘\(\varvec{\tilde{\mathbf{u}}}_\ell \bar{s}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\) E15’ reconstruction, which uses reconstructed singular values \(\bar{s}_\ell\) per (16), is that its loss is much less sensitive to the choice of rank \({\bar{r}}_{\min }\) (for ranks greater than the actual \(r_{\min }\)), because the singular values for the higher (noisier) modes are suppressed. In contrast, the loss of the standard ‘\(\varvec{\tilde{\mathbf{u}}}_\ell {\tilde{s}}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\)’ approach increases (dramatically in some cases) for ranks \(r > r_{\min }\). Thus, we will see in Sect. 3 that the ‘E15’ method achieves lower loss and less variability than the standard ‘\(\varvec{\tilde{\mathbf{u}}}_\ell {\tilde{s}}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\)’ method.

Figure 2c also shows the \(\varvec{\bar{\mathbf{A}}}_r = \sum _{\ell =1}^r \varvec{\tilde{\mathbf{u}}}_\ell s_\ell c_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\) reconstruction loss, which we show in “Appendix 2” to be the minimum possible loss if reconstructing with the noisy singular vectors. Note that the ‘E15’ loss nearly overlays on this theoretical minimum.

Reconstruction methods from the prior literature result in higher losses than that achieved herein. Raiola et al. (2015) suggest the \(\varvec{\tilde{\mathbf{u}}}_\ell {\tilde{s}}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\) reconstruction at rank \(r_F = 12\) (per 6) or \(r_{{\tilde{F}}} = 19\) (per 7). Epps and Techet (2010) suggest rank \(r = 3\). The ELF method can not be used in this example, because the CFD grid was unstructured, and the 2D DCT-II step in the ELF method requires data on a plaid grid.

2.2 Outline

The following sections provide examples that illustrate additional details of the proposed methods. In Sect. 3, we consider analytic examples in order to provide a “controlled environment” within which to present and validate some finer details of the present theory. In Sects. 4 and 5, we consider application to synthetic PIV data, which raises “real world” issues yet still affords us the ability to validate our results using clean data. Finally, in Sect. 6, we demonstrate the methods on real PIV data.

3 Analytic examples and discussion

3.1 Analytic minimal working example

Fig. 3
figure 3

(Sect. 3.1 example) Monte Carlo results: a singular values showing approximation of the noisy singular values \(\langle {\tilde{s}}_k\rangle\) (filled symbols) and reconstruction of the clean singular values \(\langle \bar{s}_k\rangle\) (open symbols). b Root mean square error of the spatial modes. c, d Mean and standard deviation of the reconstruction loss. Loss estimate (22) is marked “\({\bar{\varDelta }}_r\) approx.”. The symbols marked \(r_B\), \(r_F\), and \(r_{{\tilde{F}}}\) indicate the Brindise ELF method and the Raiola criteria (6) and (7), respectively. Here, \(T=40\), \(D=100\), \(\epsilon = 10^{-3}\), \(\epsilon ' = 1.30\epsilon\), \(\epsilon '' = 0.97\epsilon\), and \({\bar{\epsilon }} = 1.18\epsilon\)

Consider data constructed via \(\mathbf{A}= \mathbf{U}\, \mathbf{S} \, \mathbf{V}^\intercal\), with

$$\begin{aligned} \begin{aligned} U_{tk}&= \cos \left( \tfrac{2\pi }{T}(t-1)(k-1) - \tfrac{\pi }{4}\right) \sqrt{2/T} \\ S_{kk}&= 10^{-4(k-1)/(T-1)} \\ V_{dk}&= \sin \left( \tfrac{\pi }{D+1} d \, k\right) \sqrt{2/(D+1)} ~ , \end{aligned} \end{aligned}$$

with \(T=40\) and \(D=100\).

In this example, a Monte Carlo simulation of \(N=10{,}000\) trials was used to evaluate the performance of various noise filtering schemes. In each trial, we generated a noisy dataset \(\varvec{\tilde{\mathbf{A}}}= \mathbf{A}+ \mathbf{E}\) (with \(\mathbf{E}\) containing i.i.d. Gaussian noise with \(\epsilon = 10^{-3}\)), made estimates \({\bar{\epsilon }}\), \(\bar{s}_k\), and \(\varvec{\bar{\mathbf{A}}}_r\), and evaluated the reconstruction loss. Figure 3 shows the mean and standard deviation of the results for the N trials.

3.1.1 Estimates of the clean singular values

First, consider estimates of the clean singular values. Figure 3a shows good agreement between the clean singular values (black curve) and the ‘E15’ reconstruction \(\bar{s}_k\) (16) (green dashed curve). The reason for this good agreement is that the noisy singular values (red curve) are well approximated by \({\tilde{s}}_k \approx \sqrt{s_k^2 + (\epsilon '{\hat{s}}_k)^2}\) (green dashed curve, labeled ‘\(\langle {{\tilde{s}}}_k\rangle\) E15’), which is the theoretical basis of the ‘E15’ reconstruction. This approximation works well here, because the clean singular values decay rapidly as compared to the noisy ones.

A more theoretically-sophistocated estimate of the clean singular values will be referred to as the ‘EK18’ reconstruction

$$\begin{aligned} \bar{s}_k = {\left\{ \begin{array}{ll} \tfrac{1}{2}\big ({\tilde{s}}_k + \sqrt{{\tilde{s}}_k^2 - 2{\bar{\epsilon }}^2 D}\, \big ) &{} \quad ( k < k_c) \\ 0 &{} \quad (k_c \le k ) \end{array}\right. } , \end{aligned}$$

where cutoff index \(k_c\) is the minimum k such that \({\tilde{s}}_k < \text {max}\{ {\bar{\epsilon }}\sqrt{2D}, ~ {\bar{\epsilon }}(\sqrt{D}+\sqrt{fT})\}\) (Krivitzky and Epps 2017). This cutoff ensures both that (19) yields a real number and that perturbation theory is accurate. Equation (19) follows from perturbation theory, which predicts \({\tilde{s}}_k \approx s_k + \tfrac{1}{2}\epsilon ^2 D /s_k\) (Epps and Krivitzky 2019). For the Sect. 3.2 example (Fig. 4a), neither reconstruction is perfect, but again the simple ‘E15’ method (16) works well.

3.1.2 Root mean square error (rmse)

In order to help orient the reader as to the levels of noise in the modes, we find it useful to define the following mode indices:

$$\begin{aligned} \begin{aligned} k_F = \text {min }k\text { such that } {\tilde{s}}_k&< {\bar{\epsilon }}\sqrt{TD} \\ k_2 = \text {min }k\text { such that } {\tilde{s}}_k&< {\bar{\epsilon }}(\sqrt{D}+\sqrt{fT}) \\ k_\epsilon = \text {min }k\text { such that } {\tilde{s}}_k&< {\bar{\epsilon }}\sqrt{D} \end{aligned} ~ . \end{aligned}$$

Since \(\text {rmse}(\varvec{\tilde{\mathbf{v}}}_k) \approx {\bar{\epsilon }}/{\tilde{s}}_k\), these three indices correspond to modes with \(\text {rmse}(\varvec{\tilde{\mathbf{v}}}_k) \approx 1/\sqrt{TD}, 1/(\sqrt{D}+\sqrt{fT})\), and \(1/\sqrt{D}\), respectively. Moreover, index \(k_F\) marks the first mode that fails the (Epps and Techet 2010) criterion (5), \(k_2\) provides a rough approximation for the minimum-loss reconstruction rank, and \(k_\epsilon\) is the minimum index for which the noisy singular values overlay on the Marchekno–Pastur noise distribution.

Figure 3b shows the root mean square error of the spatial modes, comparing the theoretical prediction (13) to numerical results. Here, the theory slightly over-predicts the rmse, because the rmse is proportional to the noise level \({\bar{\epsilon }}\), and the estimated \({\bar{\epsilon }}= 1.18\epsilon\) is too large. Nevertheless, the form of the rmse is correct, and the optimum index \({\bar{r}}_{\min }\) (15) is predicted within \(\pm \,2\) indices of what it would be if \({\bar{\epsilon }}\) was more accurate.

3.1.3 Reconstruction loss

Figure 3c shows the reconstruction loss for the ‘E15’ method (16)/(17) and the ‘EK18’ method (19)/(17) with and without \({\bar{c}}_\ell\) (41). For reference, we also show the \(\varvec{\bar{\mathbf{A}}}_r = \sum _{\ell =1}^r \varvec{\tilde{\mathbf{u}}}_\ell s_\ell c_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\) and \(\varvec{\bar{\mathbf{A}}}_r = \sum _{\ell =1}^r \varvec{\tilde{\mathbf{u}}}_\ell {\tilde{s}}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\) methods, which are the best- and worse-case scenario reconstructions. The variable \(c_\ell\) is derived in “Appendix 2”, where we show that among all reconstructions that use the noisy singular vectors, the reconstruction that minimizes the loss \(\varDelta _r\) is the one with \(\bar{s}_\ell = s_\ell c_\ell\), where

$$\begin{aligned} \begin{aligned} c_\ell&\equiv \sum _{k=1}^T (\varvec{\tilde{\mathbf{u}}}_\ell \cdot \varvec{{\mathbf{u}}}_k) \, (\varvec{\tilde{\mathbf{v}}}_\ell \cdot \varvec{{\mathbf{v}}}_k) \, s_k/s_\ell ~ . \end{aligned} \end{aligned}$$

This \(c_\ell\) accounts for the projections of the noisy singular vectors in each of the clean singular vector directions.

The inset of Fig. 3c shows that (for all five methods) the loss is very sensitive to choice of rank for \(r < k_2\) but is relatively insensitive to rank for \(r>k_2\). The reason for this behavior is that modes \(k<k_2\) contain most of the signal content, whereas modes \(k>k_2\) contain mostly noise.

Note that for the ‘\(\varvec{\tilde{\mathbf{u}}}_\ell \bar{s}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\) E15’ method, the rank \({\bar{r}}_{\min } = 19\) (from 15) yields \(\varDelta _{{\bar{r}}_{\min }} = 2.60\times 10^{-3}\), which is just 2.5% larger than the minimum of the ‘E15’ loss curve \(\varDelta _{r_{\min }} = 2.53\times 10^{-3}\), where \(r_{\min }=21\). Also, note that at \({\bar{r}}_{\min }\) the ‘E15’ loss is nearly equal to the loss of the ‘\(\varvec{\tilde{\mathbf{u}}}_\ell s_\ell c_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\)’ method, which is the theoretical minimum.

As shown in Fig. 3d, the ‘E15’ method also has the least variation in loss, almost as low as the hypothetical ‘\(\varvec{\tilde{\mathbf{u}}}_\ell s_\ell c_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\)’ method.

3.1.4 Estimate of reconstruction loss

One motivating question of this article is What is the limit to the accuracy of SVD-based data reconstruction? We can answer this question mathematically with the following estimate of the loss:

$$\begin{aligned} \boxed { {\bar{\varDelta }}_r = {\bar{\epsilon }}^2 Dr ~ + \sum _{k=r+1}^T \bar{s}_k^2 } ~ , \end{aligned}$$

where the \(\bar{s}_k\) are the reconstructed singular values (16). Equation (22) is an approximation of the hypothetical loss prediction (61) derived in “Appendix 3”. This approximation (22) has the correct asymptotic behavior at \(k \approx 1\) and correctly equals \({\bar{\epsilon }}^2 TD\) at \(k=T\) (since \(\epsilon ^2 T D\) is the loss of the original noisy data). Figure 3c shows this “\({\bar{\varDelta }}_r\) approx.” is reasonable; at rank \({\bar{r}}_{\min }\), Eq. (22) predicts \({\bar{\varDelta }}_{{\bar{r}}_{\min }} = 2.89\times 10^{-3}\), which is just 11% higher than the actual ‘E15’ loss \(\varDelta _{{\bar{r}}_{\min }}\).

Clearly, the loss predicted by Eq. (22) has a minimum, since \(\sum _{k=r+1}^T \bar{s}_k^2\) monotonically decreases with r whereas the \({\bar{\epsilon }}^2 Dr\) monotonically increases with r. Thus, the minimum \({\bar{\varDelta }}_r\) forms a rough limit to the accuracy of the data reconstruction. More usefully, \({\bar{\varDelta }}_{{\bar{r}}_{\min }}\) forms an estimate of the actual loss of the ‘E15’ data reconstruction.Footnote 5

3.1.5 Figure of merit for reconstruction accuracy

A figure of merit for the reconstruction can be formed by comparing the estimated reduction in loss to the loss of the original noisy data:

$$\begin{aligned} \boxed { M \equiv \frac{ {\bar{\epsilon }}^2 T D - {\bar{\varDelta }}_{{\bar{r}}_{\min }} }{ {\bar{\epsilon }}^2 T D } } ~ . \end{aligned}$$

This non-dimensional ratio can be used to gauge the improvement in accuracy.Footnote 6

The parameter M can also be loosely interpreted as the fraction of noise that has been removed. For the cylinder example of Sect. 2.1, \(M = 98\%\) (also \({\bar{\varDelta }}_{{\bar{r}}_{\min }} = 0.0034\) and \(\varDelta _{{\bar{r}}_{\min }} = 0.0031\)). For the present analytic example (Sect. 3.1), \(M = 48\%\).

3.1.6 Comparison to literature methods

Other reconstruction methods were examined as well. The (Brindise and Vlachos 2017) ELF method on average yields low loss, \(\langle \varDelta _r \rangle = 2.77E-3\), although it has a much larger variability in loss (\(\sigma _{\varDelta _r} = 1.09E-4\)) than the ‘E15’ method (see Fig. 4c, d). The reason for this larger variability is that the ELF method has a larger variability in rank (\(\sigma _{r_B} = 0.905\)) than the ‘E15’ method (\(\sigma _{{\bar{r}}_{\min }} = 0.494\)), so the ELF method more often chooses the rank too low, resulting in large losses.

The (Raiola et al. 2015) F and \({\tilde{F}}\) methods yield significantly higher ranks and losses than the present ‘E15’ method. For this example, the \({\tilde{F}}_k\) method always predicted \({\tilde{r}}_F = 40\), because the \({\tilde{F}}_k\) parameter was always less than 0.999. The reason that \({\tilde{F}}_k < 0.999\) here is because \({\tilde{F}}_k\) asymptotes to 1 only when the data matrix has a very high aspect ratio (\(D\gg T\)) such that the tail of the noisy singular values is nearly constant.

Not shown in the figures are the results of the (Shabalin and Nobel 2013) method, since they nearly overlay on the present ‘\(\varvec{\tilde{\mathbf{u}}}_\ell \bar{s}_\ell {\bar{c}}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\) EK18’ method, although their theoretical development and final equations differ from those presented herein.

Fig. 4
figure 4

(Sect. 3.2 example) Monte Carlo results: a singular values showing approximation of the noisy singular values \(\langle {\tilde{s}}_k\rangle\) (filled symbols) and reconstruction of the clean singular values \(\langle \bar{s}_k\rangle\) (open symbols). b Root mean square error. (c & d) Mean and standard deviation of the loss. Loss estimate (22) is marked “\({\bar{\varDelta }}_r\) approx.”. The symbols marked \(r_B\), \(r_F\), and \(r_{{\tilde{F}}}\) indicate the Brindise ELF method and the Raiola criteria (6) and (7), respectively. Here, \(T=100\), \(D=2000\), \(\epsilon = 10^{-5}\), \(\epsilon ' = 1.23\epsilon\), \(\epsilon '' = 0.96\epsilon\), and \({\bar{\epsilon }} = 1.23\epsilon\)

3.2 Effect of the distribution of singular values

In order to illustrate the effect of the distribution of singular values, consider constructing the analytic data using the singular vectors from (18) but now using the following singular values:

$$\begin{aligned} \log _{10}(s_k) = -5 \frac{k-1}{T-1} - 0.1 \sin \left( 12\pi \frac{k-1}{T-1}\right) ~ , \end{aligned}$$

The noise data were constructed by first drawing from a normal distribution with standard deviation \(\sqrt{w}\epsilon\) and then performing uniform spatial smoothing over a window of width w. This two-step process yields spatially-correlated noise with standard deviation \(\epsilon\). Results of a Monte Carlo simulation with \(\epsilon = 10^{-5}\), \(w=5\), \(T=200\), \(D=2000\), and \(N=1000\) are shown in Fig. 4.

The results in Fig. 4 generally agree with those of in Fig. 3. One key difference in the results of these examples is that here the ‘\(\varvec{\tilde{\mathbf{u}}}_\ell \bar{s}_\ell {\bar{c}}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\) EK18’ reconstruction has unacceptably poor performance in both loss (Fig. 4c) and variability (Fig. 4d). The reason is that the estimated \({\bar{c}}_\ell\) attenuates modes 56 and higher much more than the exact \(c_\ell\), because the root mean square errors of these modes are large (see Fig. 4b). Since using \({\bar{c}}_\ell\) might lead to a more lossy reconstruction than the original noisy data, we recommend not using \({\bar{c}}_\ell\) in data reconstruction.

Fig. 5
figure 5

(Sect. 4 cylinder PIV example) Generation of synthetic PIV datasets from clean CFD data

Table 1 (Sect. 4 cylinder PIV example) Results for selected PIV processing schemes

Although the two ‘EK18’ methods have a more theoretically-sophisticated foundation than the ‘E15’ method, they again are found to be less accurate and more variable than the ‘E15’ method; therefore, the ‘EK18’ methods are not recommended.

Figures 3 and 4 show that the ‘E15’ method (15)/(16)/(17) yields the least mean loss \(\langle \varDelta _r\rangle\) and the least variation \(\sigma _{\varDelta _r}\) of all the practical methods shown. Moreover, the loss from the ‘E15’ method is very close to that from the theoretical minimum-loss method ‘\(\varvec{\tilde{\mathbf{u}}}_\ell s_\ell c_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\)’. Here, the optimum reconstruction rank is predicted to be \({\bar{r}}_{\min } = 66\), while the actual minimum of the ‘E15’ loss curve occurs at \({r}_{\min } = 69\). However, the ‘E15’ loss curve is very flat, and the loss \(\varDelta _{{\bar{r}}_{\min }} = 1.50 \times 10^{-5}\) is just 0.9% higher than \(\varDelta _{{r}_{\min }} = 1.48 \times 10^{-5}\).

Again, Eq. (22) provides an acceptable estimate of the loss. Here, (22) predicts \({{\bar{\varDelta }}}_{{\bar{r}}_{\min }} = 2.08 \times 10^{-5}\), about 40% larger than the actual \(\varDelta _{{\bar{r}}_{\min }}\). Here, the reconstruction merit (23) is \(M = 32\%\).

4 Synthetic PIV: cylinder flow

4.1 Generation of datasets

Fig. 6
figure 6

(Sect. 4 cylinder PIV example) Singular values, root mean square error, and reconstruction loss. In a, threshold levels are shown with \(\epsilon = 1.90 \times 10^{-5}\), and the M–P fit is made with \(\epsilon ' = 1.87 \times 10^{-5}\) and \(f=7.1\). In b, \({\bar{r}}_{\min } = 16\) is the index for which the rmse is within 5% of the ceiling \(\sqrt{2/D}\). In c, \(r_{\min } = 15\) is the index for minimum ‘\(\tilde{\mathbf{u}}_\ell {\bar{s}}_\ell \tilde{\mathbf{v}}_\ell ^\intercal\) E15’ loss

In this section, we consider a synthetic PIV data set generated using Sect. 2.1 CFD data. As illustrated in Fig. 5, a clean image pair was created for each time step, with particles placed randomly in the initial image, and then advected with the local flowfield (as interpolated from the clean CFD data) for the second image. The noisy image color intensity was obtained by reducing that of the corresponding clean image and adding uniformly-distributed noise to each pixel. PIV processing was performed in DaVis 8.3.1, using several multi-pass approaches as summarized in Table 1. For the figures, PIV processing was performed with an initial interrogation window size of \(64\times 64\) px with 50% overlap, then three passes on a final window size of \(32\times 32\) px with 75% overlap. Since the noisy data at the edge of the PIV domain have non-Gaussian error, these data were cropped prior to SVD analysis. The resulting vector field contained \(126\times 84\) vectors (\(D = 21,168\)) and \(T = 455\) timesteps. Further details regarding generation and processing of the synthetic PIV images are given in (Epps and Krivitzky 2019).

Here, we introduce the following notation: \(\mathbf{A}\phantom {\!^*}\) = clean CFD data, \(\mathbf{A}\!^*\) = clean PIV data, \(\varvec{\tilde{\mathbf{A}}}\!^*\) = noisy PIV data, and \(\varvec{\bar{\mathbf{A}}}\!^*\) = reconstructed PIV data (which should agree with \(\mathbf{A}\!^*\)). We also redefine the reconstruction loss from (4) as \(\varDelta _r \equiv \Vert \mathbf{A}\!^* - \varvec{\bar{\mathbf{A}}}^*_r \Vert _F^2\).

4.2 Results overview

As in previous examples, the proposed methods prove successful:

  • The error estimation procedure of “Appendix  1” produces an accurate estimate \({\bar{\epsilon }}= 1.85 \times 10^{-5}\) of the actual rms error \(\epsilon = 1.90 \times 10^{-5}\), which is the rms difference between the clean and noisy PIV data.

  • Equation (16) yields an accurate reconstruction of the clean singular values for \(k < {\bar{r}}_{\min }\), which are the important modes (see Fig. 6a).

  • The ‘E15’ noise filtering method (16)/(17) yields losses nearly as low as the theoretical minimum-loss ‘\(\tilde{\mathbf{u}}_\ell s_\ell c_\ell \tilde{\mathbf{v}}_\ell ^\intercal \,\)’ curve (see Fig. 6c).

  • The Brindise method yields rank and loss similar to those of the present approach, whereas the Raiola methods yield much higher rank and loss.

  • The estimated loss \({\bar{\varDelta }}_{{\bar{r}}_{\min }} = 3.0\times 10^{-4}\) from (22) is within a factor of 2 of the actual \({\varDelta }_{{\bar{r}}_{\min }} = 1.6\times 10^{-4}\).

  • Here, the reconstruction merit (23) is \(M = 91\%\).

Figure 7 shows a snapshot of the vorticity field of the clean PIV data, noisy PIV data, ‘E15’ reconstruction at \({\bar{r}}_{\min } = 16\), and the standard ‘\(\tilde{\mathbf{u}}_\ell {\tilde{s}}_\ell \tilde{\mathbf{v}}_\ell ^\intercal \,\)’ reconstruction at \(r = k_F -1 = 7\). Note that while the original noisy-data vorticity field contains significant noise, the character of the vorticity field is restored by the ‘E15’ reconstruction. For example, the ‘E15’ reconstruction recovers the thin blue ‘stripe’ in the middle of the frame and the green ‘spike’ near the right side. Choosing a lower rank, such as \(r = 7\), results in additional smoothing but loss of some of these fine details of the flowfield.

4.3 On the use of \(w=1\) to predict \({\bar{r}}_{\min }\)

The important new feature of this example is that it uses PIV data, which contain spatially-correlated noise. Consequently, the noisy singular values in Fig. 6a are best fit by at Marchenko–Pastur distribution that has spatial-correlation parameter \(f > 1\) (here, \(f = 7.1\)). Inserting this \(f = 7.1\) into (12) yields an effective smoothing window width of \(w=13.7\), which is reasonable considering that the data were smoothed three times using the 9 nearest neighbors. Also, the data contain spatial correlation due to being acquired in overlapping interrogation windows.

A question arises as to whether the value \(w = 13.7\) should be used to estimate the root mean square error (rmse) of the SVD modes (and subsequently the optimal reconstruction rank). Figure 6b compares the rmse predictions (13) using both \(w=13.7\) and \(w=1\), as well as the numerically-computed rmse (which is that between the noisy and clean PIV modes). The optimal reconstruction rank (15) predicted using the \(w=1\) rmse is marked \({\bar{r}}_{\min } = 16\), and an alternative rank (predicted using the \(w=13.7\) rmse) is marked \({\bar{r}}_{\text {alt}} = 13\). In this case, both ranks are nearly the same as the actual rank \({r}_{\min } = 15\) that yields minimum ‘E15’ loss. Moreover, the difference in loss between these three ranks is negligible, so use of \(w=13.7\) or 1 is somewhat of a moot point in this example.

However, in other examples (Sects. 5, 6), we have found that using the rmse predicted with \(w=1\) yields a more accurate estimate of the optimal reconstruction rank. Using \(w=1\) drives the rmse down, which drives the \(w=1\) rank \({\bar{r}}_{\min }\) lower than the \(w>1\) rank \({\bar{r}}_{\text {alt}}\). Considering the shape of the loss curve (see Fig. 6c), it is much better to estimate the rank too high than too low. Therefore, it is recommended that \(w=1\) should be used to predict rmse and subsequently \({\bar{r}}_{\min }\).

Fig. 7
figure 7

(Sect. 4 cylinder PIV example) Vorticity field for a single timestep, illustrating the ‘clean PIV’ and ‘noisy PIV’ data, as well as two reconstructions: the \(\tilde{\mathbf{u}}_\ell {\tilde{s}}_\ell \tilde{\mathbf{v}}_\ell ^\intercal \,\) reconstruction at rank \(r=k_F-1 =7\), and the \(\tilde{\mathbf{u}}_\ell {\bar{s}}_\ell \tilde{\mathbf{v}}_\ell ^\intercal \,\) E15 reconstruction at rank \({\bar{r}}_{\min } = 16\)

4.4 Effect of PIV processing scheme

Another question arises as to the efficacy of the proposed noise filtering method with respect to different PIV processing schemes. Table 1 compares results for several PIV processing approaches. Interestingly, the minimum loss \(\varDelta _{r_{\min }}\) decreases as the final interrogation window size increases. This result is explained by the fact that \(\varDelta _{r_{\min }}\) is some fraction of \(\epsilon ^2 TD\), and \(\epsilon\) also decreases with increasing interrogation window size.

For all cases in Table 1, the tail fit method of “Appendix  1” produces an accurate estimate \({\bar{\epsilon }}\) of the actual error \(\epsilon\), and the rank \({\bar{r}}_{\min }\) (predicted using \(w=1\)) is very close to the actual optimum rank \(r_{\min }\). Moreover, the resulting loss \(\varDelta _{{\bar{r}}_{\min }}\) is nearly as low as the minimum loss \(\varDelta _{r_{\min }}\) for most cases. Again, the Brindise ELF method yields slightly higher losses, and the Raiola F and \({\tilde{F}}\) methods yield significantly higher losses.

5 Synthetic PIV: channel flow

In this example, we consider synthetic PIV of a much more complex flowfield, fully-developed turbulent channel flow. This example provides further validation of the proposed methods, and it allows us to illustrate the effects of sampling timestep and number of snapshots on the proposed methods.

5.1 Generation of data sets

Synthetic PIV images were generated as in Sect. 4, with particles placed randomly in the initial image and advected with the local flowfield for the second image. Here, the flowfield was interpolated from direct numerical simulation (DNS) data of turbulent channel flow (Kim et al. 1987; Lee et al. 2013) obtained from the Johns Hopkins turbulence database (Li et al. 2008; Graham et al. 2016). The image size was \(1024\times 1024\) px, with 20,480 particles (\(\approx 20\) particles per \(32\times 32\) px interrogation window). Image pairs were generated corresponding to DNS domain \(0 \le x \le 1\), \(-1 \le y \le 0\) and the first 2000 timesteps of the DNS database.

PIV processing was performed in DaVis 8.3.1, using an initial interrogation window size of \(64\times 64\) px with 50% overlap, then three passes on a final window size of \(32\times 32\) px with 75% overlap. After cropping the edges, the resulting vector field contained \(126\times 126\) vectors (\(D = 31,752\)) and 2000 timesteps.

Fig. 8
figure 8

(Sect. 5 Channel flow PIV example) Singular values, rms error, and reconstruction loss

5.2 Baseline case (\(\delta t, T=500\))

In this subsection, we consider only the first \(T=500\) timesteps of PIV data (spaced \(\delta t\) apart, where \(\delta t\) is the DNS database timestep). Here, the rms error between the noisy and clean PIV data is \(\epsilon =0.0105\), and the Marchenko–Pastur tail fit predicts \({\bar{\epsilon }}= 0.0109\) and \(f=7.9\) (\(w=15.3\)).

The key difference between this turbulent flow example and the previous ones is that here, the clean singular values decay much more slowly. As a result, there is less separation between the clean and noisy PIV singular values (see Fig. 8a). For example, here \({\tilde{s}}_T^*/s_T^* = 7.7\), whereas in the PIV cylinder example \({\tilde{s}}_T^*/s_T^* = 15.3\), and in the analytic examples of Sects. 3.2 and 3.1, \({\tilde{s}}_T/s_T = 32\) and 49, respectively.

This reduced separation between the clean and noisy singular values makes it challenging to correctly estimate the error level \({\bar{\epsilon }}\) and effective smoothing window width w.Footnote 7 As a result, the rmse predictions end up too large when using \(w>1\) (c.f. compare the \(w=15.3\) theory curve and the numeric rmse curve in Fig. 8b). Instead, the rmse should be estimated using \(w=1\), which provides a conservative estimate of the rmse that is useful for predicting the optimum reconstruction rank \({\bar{r}}_{\min }\).

Figure 8c shows the reconstruction loss for this channel flow case: The estimated reconstruction rank \({\bar{r}}_{\min }=105\) yields a loss of \(\varDelta _{{\bar{r}}_{\min }} = 497\), which is nearly as low as the minimum ‘E15’ loss \(\varDelta _{r_{\min }} = 495\) (at \(r_{\min } = 114\)). As in previous examples, the Brindise ELF method has slightly higher loss (\(\varDelta _{r_B} = 569\) with \(r_B = 110\)), and the Raiola methods have higher losses (\(\varDelta _{r_{{\tilde{F}}}} = 857\) at \(r_{{\tilde{F}}} = 180\)). Equation (22) again provides a very good loss estimate: \({\bar{\varDelta }}_{{\bar{r}}_{\min }} = 494\) (just 0.8% lower than \(\varDelta _{{\bar{r}}_{\min }}\)). The reconstruction merit (23) is \(M = 74\%\).

Fig. 9
figure 9

(Sect. 5.3 Channel flow PIV example) Effect of timestep and number of snapshots

5.3 Effect of timestep and number of snapshots

Two important questions to address at this point are: (1) how do the various noise filtering methods perform as the separation between the noisy and clean singular values varies?, and (2) how do the timestep and number of frames in the dataset affect the singular values? To address these questions, we performed the SVD on six datasets subsampled with various timesteps (\(\delta t\), \(2\delta t\), and \(4\delta t\), where \(\delta t\) is the DNS database timestep) and number of snapshots (\(T=500, 1000, 2000\)).

Figure 9a shows the clean singular values for these six cases. When plotted versus normalized mode index k / T, the cases with the same timestep (but different T) overlay, indicating that the number of snapshots T has little effect on the clean singular values. Rather, the shape (slope and magnitude) of the tail of the clean singular values is dictated by the timestep.

By contrast, the noisy singular values are relatively insensitive to the timestep (Fig. 9b), so a larger timestep causes there to be less separation between clean and noisy singular values.

Figure 9c shows the loss for the three \(T=500\) cases (with the \(\delta t\) case carried over from Sect. 5.2). As expected, a larger timestep results in more loss, and less of a “bucket” below the noisy dataset loss \(\epsilon ^2 TD\) within which to advantageously filter noise. Consequently, the reconstruction merit (23) decreases with increasing timestep; for the \(\delta t\), \(2\delta t\), and \(4\delta t\) cases, we find \(M = 74\%\), 54%, and 9%, respectively. In all three cases, the present methods yield reasonable predictions of the optimal reconstruction rank \({\bar{r}}_{\min }\). The ELF method performs well at \(\delta t\) and \(2\delta t\) but not \(4\delta t\). In contrast, the Raiola method performs best at \(4\delta t\).

For reference, the rmse for the {\(2\delta t\), \(T=500\)} case are shown in Fig. 9d. Here, the effect of using \(w>1\) versus \(w=1\) is very pronounced, with the \(w=1\) rmse predictions yielding a much better optimal rank estimate \({\bar{r}}_{\min }=164\). The \(w>1\) rmse predictions yield \({\bar{r}}_{\text {alt}}=111\), which results in significantly more loss than using \({\bar{r}}_{\min }\).

5.4 Comment regarding optimal rank criterion

Some readers might find it controversial to use the root mean square error of the spatial modes (rmse\((\varvec{\tilde{\mathbf{v}}}_k\))) in order to predict the optimal reconstruction rank. The reasoning is that modes with very similar energy might switch order, so their computed rmse might be very large, even if the modes themselves did not change.

Regarding modes with similar energy (i.e. small gap between neighboring singular values), there are two important cases to consider: First, consider “paired modes”, which are modes with similar energy that are well below the noise ceiling. Equation (13) shows that the rmse of each of these modes will indeed be higher than the rmse of “isolated modes” (i.e. those with well-separated singular values). The reason is that paired modes can and do mix upon perturbation. However, Wedin’s theorem ensures that the subspace spanned by these modes, as a group, is only slightly perturbed. Thus, paired modes below the noise ceiling should be and are used in our data reconstruction method.

The second case to consider is “noise modes”, which are modes with rmse at the noise ceiling (rmse\((\varvec{\tilde{\mathbf{v}}}_k) \approx \sqrt{2/D}\)). These modes have similar energy, since their singular values fall in the tail of the distribution. In (Epps and Krivitzky 2019), we show that modes with rmse\((\varvec{\tilde{\mathbf{v}}}_k) \approx \sqrt{2/D}\) constitute random noise. Thus, these modes should be discarded for noise filtering.

A subtle point is that Eq. (15) keeps all the paired modes and only discards the noise modes, because \({\bar{r}}_{\min }\) is set as the “maximum k such that \(t_k > 5\%\)”. In other words, there might be some paired modes with \(t_k > 5\%\), but as long as there are some isolated modes with \(t_k < 5\%\) for larger k, then those paired modes are kept. For example, note in Fig. 9d that modes k = 134–135 and 142–143 have rmse above the 5% cutoff, but these paired modes are kept, because there are other isolated modes between those and \({\bar{r}}_{\min } = 164\), which is the last mode with rmse below the 5% cutoff.

6 Laminar jet PIV example

In this final example, we apply the present noise filtering methods to a real PIV dataset. Here, we consider a publicly-available PIV dataset that captures the Kelvin–Helmholz instabilities of a laminar jet in a quiescent fluid (Neal et al. 2015). This set of PIV images is unique in that it is accompanied by ‘clean’ vector fields that have been obtained using high-dynamic-range PIV (PIV-HDR), which has higher accuracy than standard PIV because it uses multiple cameras with very high image resolution (Neal et al. 2015).

6.1 Generation of datasets

A sample PIV image is shown in Fig. 10; the red solid rectangle highlights the cropped region used for PIV processing, and the green dash-dot rectangle represents the region where the PIV-HDR data are available. Since the PIV and PIV-HDR data are not available on the same grid, the SVDs of these two datasets cannot be compared directly, and the reconstruction loss must be computed only over the PIV-HDR region. For the reconstruction loss, the PIV-HDR data are down-sampled to the standard PIV grid locations. PIV processing was performed using DaVis 8.3.1, with a final window size of \(16 \times 16\) px with 75% overlap, leading to a \(43 \times 84\) vector grid. Figure 10 displays a sample vector field and corresponding vorticity contours; flow is left to right.

Fig. 10
figure 10

(Sect. 6 Laminar Jet PIV example). (Left) Sample PIV image (Neal et al. 2015), with cropped region used for PIV processing (red box), and region where the PIV-HDR data are available (green dash-dot box). (Right) Resulting PIV velocity vectors (every fourth vector shown) and vorticity contours

Fig. 11
figure 11

(Sect. 6 Laminar Jet PIV example) Singular values, rmse, and reconstruction loss

Fig. 12
figure 12

(Sect. 6 Laminar Jet PIV example) Snapshots of vorticity (with time-mean flow removed). The small panels show the noisy data and ‘\(\varvec{\tilde{\mathbf{u}}}_\ell \bar{s}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\) E15’ reconstruction (\({\bar{r}}_{\min } = 30\)) at two times (\(t = 81\) and 161). The large panels blow up the PIV-HDR region at time \(t=161\), showing the noisy PIV flowfield, E15 reconstruction, and ‘clean’ PIV-HDR data

6.2 Results and discussion

Similar to the previous example, singular values of the noisy data (Fig. 11a) are very-well fit by a Marchenko–Pastur distribution with \(f>1\) (here \(f = 8.1\), \(w=15.7\)). In this example, a long ‘transition region’ \(k_F< k < k_\epsilon\) exists, wherein the modes transition from fairly clean (\(k<k_F = 3\)) to complete noise (\(k>k_\epsilon = 125\)). With the ‘E15’ reconstruction (16), the singular values in this transition region are reduced. Although ‘clean’ data exists, their singular values are not directly comparable to those of the PIV dataset due to the different region sizes, so ‘clean’ singular values are not shown.

Reconstruction loss (Fig. 11c) was determined by sampling both the ‘clean’ PIV-HDR data and the PIV reconstruction datasets at points within the HDR region. The results generally follow the previous examples: The standard ‘\(\varvec{\tilde{\mathbf{u}}}_\ell {\tilde{s}}_\ell \varvec{\tilde{\mathbf{v}}}_\ell ^\intercal\)’ reconstruction has a minimum (here, \(\varDelta _r = 377\) at \(r=18\)) and then rises for higher ranks. The ELF method selects \(r_B=15\) modes for reconstruction, resulting in a loss of \(\varDelta _{r_B} = 395\). The Raiola \({\tilde{F}}\) method selects rank \(r_{{\tilde{F}}} = 43\), yielding loss \(\varDelta _{r_{{\tilde{F}}}} = 430\). The ‘E15’ reconstruction selects rank \({\bar{r}}_{\min } = 30\) and has loss \(\varDelta _{{\bar{r}}_{\min }} = 360\), which happens to be the minimum E15 loss. Of the three methods, again the E15 reconstruction has the minimum loss. The Raiola F method was not applicable, because the clean singular values were unknown for this real PIV dataset. The loss estimate (22) and reconstruction merit (23) also are not applicable here, because the loss is not computed over the entire PIV domain.Footnote 8

Snapshots of vorticity for the noisy data compared to the ‘E15’ reconstruction (at \({\bar{r}}_{\min } = 30\)) are shown on the left of Fig. 12 at two different times for the entire PIV region. The time-mean velocity field has been removed, revealing significant noise in the measurements of unsteady components. The E15 reconstruction filters out much of the noise, leaving a much clearer view of the important flow structures. The three large panels of Fig. 12 show the noisy data, E15 reconstruction, and ‘clean’ PIV-HDR data on the HDR region. The E15 reconstruction matches the prominent features of the ‘clean’ data quite well.

7 Conclusions

This paper addresses several questions regarding noise filtering via the singular value decomposition:

How effective is the SVD for filtering out the noise and reconstruction the clean data?

The effectiveness of the SVD for noise filtering is dependent on the decay rate of the clean singular values and the noise level of the data. Recall, the reconstruction loss was reasonably estimated by Eq. (22), which predicted

$$\begin{aligned} {\bar{\varDelta }}_{{\bar{r}}_{\min }} = {\bar{\epsilon }}^2 D {\bar{r}}_{\min } ~ + \sum _{k={\bar{r}}_{\min }+1}^T \bar{s}_k^2 ~ . \end{aligned}$$

Considering (25), it is clear that the lowest losses can be achieved in problems where the optimum reconstruction rank \({\bar{r}}_{\min }\) is small and the clean singular values in the tail \(k > {\bar{r}}_{\min }\) are negligible. The examples from Sects. 2.1, 4, and 5.2 have such character, so their reconstruction loss \({\varDelta }_{{\bar{r}}_{\min }}\) was much less than the loss of the original noisy data \(\epsilon ^2 T D\).

The non-dimensional ratio

$$\begin{aligned} M \equiv \frac{ {\bar{\epsilon }}^2 T D - {\bar{\varDelta }}_{{\bar{r}}_{\min }} }{ {\bar{\epsilon }}^2 T D } ~ . \end{aligned}$$

can be used to gauge the improvement in loss. Values greater than zero suggest that the reconstruction is more accurate than the original noisy data, with values approaching unity suggesting a highly accurate reconstruction.

The character of having negligible clean singular values and a low reconstruction rank is akin to having a large separation between the tails of the clean and noisy singular values. With large separation, the filtering capacity of the SVD is quite good. However, in cases with little separation, an SVD-based reconstruction might filter some noise but not necessarily improve the accuracy (loss) over that of the original noisy data (c.f. the \(4\delta t\) curve in Fig. 9c). In Sect. 5.3, we showed that the separation between clean and noisy singular values can be increased by using a decreased timestep.

What method yields the most accurate data reconstruction, and what are the limits to its accuracy? While a number of practical reconstruction approaches were investigated, the ‘E15’ method (15)/(16)/(17) proved to be most robust, with the lowest mean loss \(\langle \varDelta _r\rangle\) and the least variation \(\sigma _{\varDelta _r}\). Moreover, choosing the rank \({\bar{r}}_{\min }\) yields nearly the minimum loss possible when reconstructing with the noisy singular vectors.

In “Appendix 3”, we show that even more accurate reconstructions could be formed if the clean singular vectors were known (or correctly estimated somehow). Thus, one focus of future work is to develop a practical method for estimating the clean singular vectors from the noisy ones.

Can the magnitude of the noise be inferred from the noisy data themselves? A method to estimate the RMS noise was presented in “Appendix  1”. The approach is to fit a Marchenko–Pasteur distribution to the tail of the singular value distribution, with the noise level and the choice of index defining the tail determined so as to minimize the least square error of this fit. This method is sufficiently robust so as to enable noise estimation (and subsequently noise filtering) in an automated data processing code.

Collectively, this body of work provides a thorough analysis of the effects of noise on the SVD of noisy data, the potential for noise estimation using the SVD, and the capabilities of the SVD for noise filtering.