Super-resolution of turbulent passive scalar images using data assimilation

Abstract

In this paper, the problem of improving the quality of low-resolution passive scalar image sequences is addressed. This situation, known as “image super-resolution” in computer vision, aroused to our knowledge very few applications in the field of fluid visualization. Yet, in most image acquisition devices, the spatial resolution of the acquired data is limited by the sensor physical properties, while users often require higher-resolution images for further processing and analysis of the system of interest. The originality of the approach presented in this paper is to link the image super-resolution process together with the large eddy simulation framework in order to derive a complete super-resolution technique. We first start by defining two categories of fine-scale components we aim to reconstruct. Then, using a deconvolution procedure as well as data assimilation tools, we show how to partially recover some of these missing components within the low-resolution images while ensuring the temporal consistency of the solution. This method is evaluated using both synthetic and real image data. Finally, we demonstrate how the produced high-resolution images can improve a posteriori analysis such as motion field estimation.

Appendices

Appendix 1: Introduction to 4DVAR algorithm

In this section, we provide a few details about variational data assimilation techniques, and more specifically the 4DVAR algorithm. For a complete methodological review of these techniques as well as applications dedicated to geophysical flow, we refer to Bennett (1992), Dimet and Talagrand (1986), Lions (1971), Vidard et al. (2000), Talagrand and Courtier (1987).

Let us assume we aim at estimating a state variable X partially known and driven by an approximately known dynamical law. In other words, we want to estimate values $X(\mathbf {x} ,t)$ for any location $\mathbf {x}$ and time $t \in [t_0,t_f]$ solution of the following system:

$$\begin{aligned} \left\{ \begin{array}{l} \dfrac{\partial X}{\partial t} + {\mathbb {M}}(X(\mathbf {x} ,t))= \nu _{d}(\mathbf {x} )\\ {\mathcal {Y}}(\mathbf {x} ,t) = {\mathbb {H}}(X(\mathbf {x} ,t)) + \nu _{o}(\mathbf {x} ,t)\\ X(\mathbf {x} ,t_0) = X_0(\mathbf {x} ) + \nu _{i}(\mathbf {x} ) \end{array} \right. \end{aligned}$$

(15)

where ${\mathbb {M}}$ is the nonlinear operator relative to the dynamics, $X_0$ is the initial vector at time $t_0$ and $(\nu _{d},\nu _{o},\nu _{i})$ are (unknown) additive control variables relative to noise on the dynamics and the initial condition respectively. In addition, noisy measurements Y of the unknown state are available through the nonlinear operator ${\mathbb {H}}$ up to $\nu _{o}$. To estimate the system’s state, a common methodology is the minimization of the cost function ${\mathcal {J}}$ :

$$\begin{aligned} {\mathcal {J}}(X)= \,& {} \dfrac{1}{2}\int _{t_0}^{t_f}\Vert \dfrac{\partial X}{\partial t} + {\mathbb {M}}(X(\mathbf {x} ,t)) \Vert _{Q^{-1}}^{2}dt \nonumber \\ \quad+\, & {} \dfrac{1}{2}\int _{t_0}^{t_f}\Vert {\mathcal {Y}}(\mathbf {x} ,t) - {\mathbb {H}}(X(\mathbf {x} ,t)) \Vert _{R^{-1}}^{2}dt \nonumber \\\quad+\, & {} \dfrac{1}{2} \Vert X(\mathbf {x} ,t_0) - X_0(\mathbf {x} ) \Vert _{B^{-1}}^{2} \end{aligned}$$

(16)

where we have introduced the information matrices R, B, Q relative to the covariance of the errors $(\nu _{d},\nu _{o},\nu _{i})$. The Mahalanobis distance that has been used reads $\Vert X \Vert _{A^{-1}} = X^TA^{-1}X$. The evaluation of X can be done by canceling the gradient $\nabla {\mathcal {J}}_X(\theta ) = lim_{\beta \rightarrow 0} \frac{{\mathcal {J}}(X + \beta \theta ) - {\mathcal {J}}(X)}{\beta }$ of (16). Unfortunately, the estimation of such gradient is in practice unfeasible for a large system’s state since it would be necessary to compute perturbations along all the components of X. One way to cope with this difficulty is to write an adjoint formulation of the problem. To that end, the adjoint variables $\lambda$ that express the errors of the dynamic model are introduced as:

$$\begin{aligned} \lambda = Q^{-1}\left( \dfrac{\partial X}{\partial t} + {\mathbb {M}}(X) \right) \end{aligned}$$

(17)

Let us now introduce a few notations

• $\frac{\partial {\mathbb {M}}}{\partial X}$ and $\frac{\partial {\mathbb {H}}}{\partial X}$ the linear tangent operators of M and H, respectively,

• $(\partial _X{\mathbb {M}})^*$ and $(\partial _X{\mathbb {H}})^*$ their adjoint operators.

It can be shown that canceling the gradient $\nabla {\mathcal {J}}_X(\theta )$ w.r.t the adjoint variables $\lambda$ leads to a retrograde integration of an adjoint evolution model that takes into account the observations. Once the adjoint variables $\lambda$ are estimated, one can recover the system state X using relation (17). Finally, recovering X leads to the following incremental algorithm:

Intuitively, the adjoint variables $\lambda$ contain information about the discrepancy between the observations and the dynamic model. They are computed from a current solution $\tilde{X}$ with the backward integration from step (ii) that encompasses both the observations and the dynamic operators. This deviation indicator between the observations and the model is then used to refine the initial condition [step (iii) and to recover the system state through an imperfect dynamic model where errors are $Q\lambda$ (step (iv)]. It should be noted that if the dynamic is perfect, the associated error covariance Q is zero and the algorithm only refines the initial condition. However from an image analysis point of view, a perfect modeling is difficult to obtain since the different models on which one can rely are usually inaccurate due, for instance, to 3D–2D projections, varying lighting conditions, completely unknown boundary conditions at the image boarders, etc.

Appendix 2: Motion estimation from a pair of images

Many possibilities are available to estimate a motion field from a pair of images. Among main strategies [correlation Keane and Adrian (1992), Lucas and Kanade (1981)] or Horn and Schunck Horn (1981), we rely in this paper on the seminal work of Horn and Schunck which aims at estimating a velocity field $\mathbf {v}$ on the image domain $\varOmega$ by the minimization of the function :

$$\begin{aligned} \mathbf {v}= & {} \min _{\mathbf {v} =(u,v)^T}\int _\varOmega \left\{ \left( \frac{\partial I(\mathbf {x} ,t)}{\partial t}+ \mathbf {v} (\mathbf {x} ,t)\cdot \nabla I(\mathbf {x} ,t)\right) ^2 \right. \nonumber \\&\left. +\, \alpha \left( |\nabla u(\mathbf {x} )|^2 + |\nabla v(\mathbf {x} )|^2\right) \right\} d\mathbf {x} , \end{aligned}$$

(18)

where $I(\mathbf {x} ,t)$ is the brightness intensity function in location $\mathbf {x}$ at time t(viewed as a continuous function). This functional is composed of an observation model (here the total derivative that assumes a global conservation of $I$ along time: ${\rm d}I/{\rm d}t \sim 0$) and a regularization term which aims at estimating velocity fields with low spatial variation of u and v components. Parameter $\alpha$ is balancing the relative influence of the observation model and the smoothing term.

Many authors have proposed alternatives (multi-resolution, optimization with various norms, ...) on the basis of Eq. (18) to design advanced models. In this paper we use the technique proposed in Chen et al. (2015) which is a combination of approaches in Cassisa et al. (2011), Corpetti et al. (2002) and aims at optimizing an observation term issued from a turbulence model for resolved scales associated with a regularization that minimizes the divergence of the flow. The model reads (with $(\bullet ) = (\mathbf {x} ,t)$):

$$\begin{aligned} \mathbf {v}= & {} \min _{\mathbf {v} =(u,v)^T}\int _\varOmega \left\{ \ \left( \frac{\partial I(\bullet )}{\partial t} + \mathbf {v} (\bullet )\cdot \nabla I(\bullet )-\frac{1}{ReSc} \Delta I(\bullet )\right) ^2 \right. \nonumber \\&\quad \left. +\, \alpha |\nabla \cdot \mathbf {v} (\bullet ))|^2\right\} d\mathbf {x} . \end{aligned}$$

(19)

All implementations details are visible in Chen et al. (2015). It should be noted that several authors have proposed motion estimation techniques for fluid flows. We refer the readers to Bruhn et al. (2005), Cassisa et al. (2011), Corpetti et al. (2002), Héas et al. (2012), Zille et al. (2014) or to the review paper Heitz et al. (2010).