Low-Rank Dynamic Mode Decomposition: An Exact and Tractable Solution

Abstract

This work studies the linear approximation of high-dimensional dynamical systems using low-rank dynamic mode decomposition. Searching this approximation in a data-driven approach is formalized as attempting to solve a low-rank constrained optimization problem. This problem is non-convex, and state-of-the-art algorithms are all sub-optimal. This paper shows that there exists a closed-form solution, which is computed in polynomial time, and characterizes the $\ell _2$-norm of the optimal approximation error. The paper also proposes low-complexity algorithms building reduced models from this optimal solution, based on singular value decomposition or eigenvalue decomposition. The algorithms are evaluated by numerical simulations using synthetic and physical data benchmarks.

Appendices

Proof of Theorem 1

We begin by showing the first part of the theorem, namely that $A_k^\star ={U}_{{\mathbf {Z}},k} {{U}_{{\mathbf {Z}},k}}^\intercal {{\mathbf {Y}}}{{\mathbf {X}}}^{\dagger }$ is a solution of (9). We first prove in this paragraph the existence of a minimizer of (9). Let us show that we can restrict our attention to a minimization problem over the set

$$\begin{aligned} {\mathcal {A}}=\{{\tilde{A}} \in \mathbb {R}^{n \times n} : \text {rank}({\tilde{A}}) \le k, \text {Im}({\tilde{A}}^\intercal ) \subseteq \text {Im}({{\mathbf {X}}})\}. \end{aligned}$$

Indeed, any matrix $A \in \{{\tilde{A}} \in \mathbb {R}^{n \times n} : \text {rank}({\tilde{A}}) \le k\}$ can be decomposed in two components: $A= A^\parallel + A^\perp$ where $A^\parallel$ belongs to the set ${\mathcal {A}}$, such that columns of $A^\parallel$ are orthogonal to those of $A^\perp$, i.e., $A^\perp ( A^\parallel )^\intercal =0$. From this construction, we have that rows of $A^\perp$ are orthogonal to rows of ${{\mathbf {X}}}$. Using this decomposition, we thus have that $\Vert {{\mathbf {Y}}}- A {{\mathbf {X}}}\Vert _F^2=\Vert {{\mathbf {Y}}}- A^\parallel {{\mathbf {X}}}\Vert _F^2$. Moreover, because of this orthogonal property, we have that $\text {rank}( A)=\text {rank}( A^\parallel ) +\text {rank}( A^\perp )$ so that $\text {rank}( A^\parallel ) \le \text {rank}( A)$. In consequence, if A is a minimizer of (9), then $A^\parallel$ is also a minimizer since it leads to same value of the cost function and since it is admissible: $\text {rank}( A^\parallel ) \le \text {rank}( A) \le k$. Therefore, it is sufficient to find a minimizer over the set ${\mathcal {A}}$.

Now, according to the Weierstrass theorem (Bertsekas 1995, Proposition A.8), the existence is guaranteed if the admissible set ${\mathcal {A}}$ is closed and the objective function $\Vert {{\mathbf {Y}}}- A {{\mathbf {X}}}\Vert _F^2$ is coercive. Let us prove these two properties. We first show that ${\mathcal {A}}$ is closed. According to Hackbusch (2012), Lemma 2.4, the set of low-rank matrices is closed. Moreover, it is well known that a linear sub-space of a normed finite-dimensional vector space is closed (Auliac and Caby 2005, Chapter 7.2), so that the set of matrices ${\mathcal {A}}=\{{\tilde{A}} \in \mathbb {R}^{n \times n} : \text {Im}({\tilde{A}}^\intercal ) \subseteq \text {Im}({{\mathbf {X}}})\}$ is closed. Since ${\mathcal {A}}$ is the intersection of two closed sets, we deduce that ${\mathcal {A}}$ is closed. Next, we show coercivity. Let us consider the SVD of any $A\in {\mathcal {A}}$: $A=U_A\varSigma _A V_A^\intercal$, where $\varSigma _A=\text {diag}(\sigma _{A,1}\cdots \sigma _{A,k})$. From the definition of the Frobenius norm, we have for any $A \in {\mathcal {A}}$, $\Vert A\Vert _F =( \sum _{i=1}^k\sigma _{A,i}^2)^{1/2}$. We have that $\Vert A\Vert _F \rightarrow \infty$ if a non-empty subset of singular values, say $\{\sigma _{A,j}\}_{j \in {\mathcal {J}}}$, tend to infinity. Therefore, we have

$$\begin{aligned} \lim _{ \Vert A\Vert _F \rightarrow \infty :A \in {\mathcal {A}} } \Vert {{\mathbf {Y}}}- A {{\mathbf {X}}}\Vert _F^2&= \lim _{\Vert A\Vert _F \rightarrow \infty : A \in {\mathcal {A}} } \Vert {{\mathbf {Y}}}\Vert ^2_F -2 \,\text {trace}(Y^\intercal A {{\mathbf {X}}})+ \Vert A {{\mathbf {X}}}\Vert _F^2, \\&= \lim _{ \Vert A\Vert _F \rightarrow \infty :A \in {\mathcal {A}} } \Vert A {{\mathbf {X}}}\Vert _F^2 = \lim _{\Vert A\Vert _F \rightarrow \infty : A \in {\mathcal {A}} } \Vert \varSigma _A V_A^\intercal {{\mathbf {X}}}\Vert _F^2, \\&= \lim _{\sigma _{A,j} \rightarrow \infty : A \in {\mathcal {A}},j \in {\mathcal {J}}} \sum _{j=1}^n \sigma _{A,j}^2 \Vert {{\mathbf {X}}}^\intercal v_A^{j} \Vert _2^2 = \infty . \end{aligned}$$

The second equality is obtained because the dominant term when $\Vert A\Vert _F \rightarrow \infty$ is the quadratic one $\Vert A {{\mathbf {X}}}\Vert _F^2$. The third equality follows from the invariance of the Frobenius norm to unitary transforms, while the last equality is obtained noticing that $\Vert {{\mathbf {X}}}^\intercal v_A^{j} \Vert _2 \ne 0$ because $v_A^{j} \in \text {Im}({{\mathbf {X}}})$ since $A \in {\mathcal {A}}$. This shows that the objective function is coercive over the closed set ${\mathcal {A}}$. Thus, using the Weierstrass theorem, this shows the existence of a minimizer of (9) in ${\mathcal {A}}$ and thus in $\{{\tilde{A}} \in \mathbb {R}^{n \times n} : \text {rank}({\tilde{A}}) \le k\}$. We will no longer restrict our attention to the domain ${\mathcal {A}}$ in the following and come back to the original problem (9) implying the set of low-rank matrices.

Next, problem (9) can be rewritten as the unconstrained minimization

$$\begin{aligned} A_k^\star \in&{{\,\mathrm{arg\,min}\,}}_{A=PQ^\intercal : P,Q \in \mathbb {R}^{n \times k}} \Vert {{\mathbf {Y}}}-A {{\mathbf {X}}}\Vert ^2_F. \end{aligned}$$

(21)

In the following, we will use the first-order optimality condition of problem (21) to characterize its minimizers. A closed-form expression for a minimizer will then be obtained introducing an additional orthonormal property. The first-order optimality condition and the additional orthonormal property are presented in the following lemma, which is proven in “Appendix B”.

Lemma 1

Problem (21) admits a solution such that

$$\begin{aligned}&P^\intercal P=I_k \end{aligned}$$

(22)

$$\begin{aligned}&{{\mathbf {X}}}{{\mathbf {Y}}}^\intercal P= {{\mathbf {X}}}{{\mathbf {X}}}^\intercal Q. \end{aligned}$$

(23)

To find a closed-form expression of a minimizer of (21), we need to rewrite condition (23). We prove that this condition is equivalent to

$$\begin{aligned} {\mathbb {P}}_{{{\mathbf {X}}}^\intercal }{{\mathbf {Y}}}^\intercal P={{\mathbf {X}}}^\intercal Q. \end{aligned}$$

(24)

Indeed, we show by contradiction that (23) implies that, for any solution of the form $PQ^\intercal$, there exists $Z\in \mathbb {R}^{m \times k}$ such that

$$\begin{aligned} {\mathbb {P}}_{{{\mathbf {X}}}^\intercal }{{\mathbf {Y}}}^\intercal P +Z={{\mathbf {X}}}^\intercal Q, \end{aligned}$$

(25)

with columns of Z in $\ker ({{\mathbf {X}}})$. Indeed, if ${\mathbb {P}}_{{{\mathbf {X}}}^\intercal }{{\mathbf {Y}}}^\intercal P +Z \ne {{\mathbf {X}}}^\intercal Q$, then by multiplying both sides on the left by ${{\mathbf {X}}}$ we obtain ${\mathbb {P}}_{{{\mathbf {X}}}}{{\mathbf {X}}}{{\mathbf {Y}}}^\intercal P +{{\mathbf {X}}}Z= {\mathbb {P}}_{{{\mathbf {X}}}} {{\mathbf {X}}}{{\mathbf {Y}}}^\intercal P \ne {{\mathbf {X}}}{{\mathbf {X}}}^\intercal Q$. Since ${\mathbb {P}}_{{{\mathbf {X}}}}$ is the orthogonal projector onto the sub-space spanned by the columns of ${{\mathbf {X}}}$, the latter relation implies that ${{\mathbf {X}}}{{\mathbf {Y}}}^\intercal P \ne {{\mathbf {X}}}{{\mathbf {X}}}^\intercal Q$ which contradicts (23). This proves that (23) implies (25).

Now, since columns of the two terms in the left-hand side of (25) are orthogonal and since columns of the matrix in the right-hand side are in the image of ${{\mathbf {X}}}^\intercal$, we deduce that the only admissible choice is Z with columns belonging both to $\ker ({{\mathbf {X}}})$ and $\text {Im}({{\mathbf {X}}}^\intercal )$, i.e., Z is a matrix full of zeros. Therefore, we obtain the necessary condition (24).

We have shown on the one hand that (23) implies (24). On the other hand, by multiplying on the left both sides of (24) by ${{\mathbf {X}}}$, we obtain (23) (${{\mathbf {X}}}{\mathbb {P}}_{{{\mathbf {X}}}^\intercal }={{\mathbf {X}}}$ because ${{\mathbf {X}}}{{\mathbf {X}}}^\dag$ is the orthogonal projector onto the space spanned by the columns of ${{\mathbf {X}}}$). Therefore, the necessary conditions (23) and (24) are equivalent.

We are now ready to characterize a minimizer of (9). According to Lemma 1, we have

$$\begin{aligned}&\min _{A \in \mathbb {R}^{n \times n} : \text {rank}(A) \le k }\Vert {{\mathbf {Y}}}- A{{\mathbf {X}}}\Vert _F^2 \nonumber \\&\quad =\min _{( {{\tilde{P}}}, {{\tilde{Q}}}) \in \mathbb {R}^{n \times k} \times \mathbb {R}^{n \times k} }\Vert {{\mathbf {Y}}}- {{\tilde{P}}}{{\tilde{Q}}}^\intercal {{\mathbf {X}}}\Vert _F^2 \quad s.t. \quad \left\{ \begin{aligned}&{{\tilde{P}}}^\intercal {{\tilde{P}}}=I_k\\&\quad {{\mathbf {X}}}{{\mathbf {Y}}}^\intercal {{\tilde{P}}} = {{\mathbf {X}}}{{\mathbf {X}}}^\intercal {{\tilde{Q}}}\\ \end{aligned}\right. , \end{aligned}$$

(26)

$$\begin{aligned}&\quad =\min _{( {{\tilde{P}}}, {{\tilde{Q}}}) \in \mathbb {R}^{n \times k} \times \mathbb {R}^{n \times k} }\Vert {{\mathbf {Y}}}- {{\tilde{P}}}{{\tilde{Q}}}^\intercal {{\mathbf {X}}}\Vert _F^2 \quad s.t. \quad \left\{ \begin{aligned}&{{\tilde{P}}}^\intercal {{\tilde{P}}}=I_k \\&\quad {\mathbb {P}}_{{{\mathbf {X}}}^\intercal }{{\mathbf {Y}}}^\intercal {\tilde{P}}={{\mathbf {X}}}^\intercal {{\tilde{Q}}}\\ \end{aligned}\right. ,\nonumber \\&\quad =\min _{ {{\tilde{P}}} \in \mathbb {R}^{n \times k} }\Vert {{\mathbf {Y}}}- {{\tilde{P}}} {{\tilde{P}}}^\intercal {{\mathbf {Y}}}{\mathbb {P}}_{{{\mathbf {X}}}^\intercal } \Vert _F^2 \quad s.t. \quad {{\tilde{P}}}^\intercal {{\tilde{P}}}=I_k,\end{aligned}$$

(27)

$$\begin{aligned}&\quad =\min _{ {{\tilde{P}}} \in \mathbb {R}^{n \times k} }\Vert ({{\mathbf {Y}}}- {{\tilde{P}}} {{\tilde{P}}}^\intercal {{\mathbf {Y}}}){\mathbb {P}}_{{{\mathbf {X}}}^\intercal } + {{\mathbf {Y}}}(I_m-{\mathbb {P}}_{{{\mathbf {X}}}^\intercal }) \Vert _F^2 \quad s.t. \quad {{\tilde{P}}}^\intercal {{\tilde{P}}}=I_k, \nonumber \\&\quad =\min _{ {{\tilde{P}}} \in \mathbb {R}^{n \times k} }\Vert {\mathbf {Z}}- {{\tilde{P}}} {{\tilde{P}}}^\intercal {\mathbf {Z}}\Vert _F^2 + \Vert {{\mathbf {Y}}}(I_m-{\mathbb {P}}_{{{\mathbf {X}}}^\intercal }) \Vert _F^2 \quad s.t. \quad {{\tilde{P}}}^\intercal {{\tilde{P}}}=I_k. \end{aligned}$$

(28)

The second equality is obtained from the equivalence between (23) and (24). The third equality is obtained by introducing the second constraint in the cost function and noticing that projection operators are always symmetric, i.e., $({\mathbb {P}}_{{{\mathbf {X}}}^\intercal })^\intercal = {\mathbb {P}}_{{{\mathbf {X}}}^\intercal },$ while the last equality follows from the definition of ${\mathbf {Z}}$ given in (15) and the orthogonality of the columns of the two terms. Problem (28) is a proper orthogonal decomposition problem with the snapshot matrix ${\mathbf {Z}}$. The solution of this proper orthogonal decomposition problem is the matrix ${U}_{{\mathbf {Z}},k}$ (with orthonormal columns) defined in Sect. 4.1, see e.g., (Quarteroni et al. 2015, Proposition 6.1). We thus obtain from (27) that

$$\begin{aligned} \min _{A \in \mathbb {R}^{n \times n} : \text {rank}(A) \le k }\Vert {{\mathbf {Y}}}- A{{\mathbf {X}}}\Vert _F^2 =\Vert {{\mathbf {Y}}}-{U}_{{\mathbf {Z}},k}{{U}_{{\mathbf {Z}},k}}^\intercal {{\mathbf {Y}}}{\mathbb {P}}_{{{\mathbf {X}}}^\intercal } \Vert _F^2 =\Vert {{\mathbf {Y}}}- {\mathbb {P}}_{{\mathbf {Z}},k} {{\mathbf {Y}}}{\mathbb {P}}_{{{\mathbf {X}}}^\intercal } \Vert _F^2 . \end{aligned}$$

(29)

Furthermore, we verify that $A_k^\star ={U}_{{\mathbf {Z}},k}{W}^\intercal$ with ${W}=({{\mathbf {X}}}^\intercal )^\dag {{\mathbf {Y}}}^\intercal {U}_{{\mathbf {Z}},k}$ is a minimizer of (21). Indeed, since ${{\mathbf {X}}}{{\mathbf {X}}}^\intercal {W}={{\mathbf {X}}}{{\mathbf {X}}}^\intercal ({{\mathbf {X}}}^\intercal )^\dag {{\mathbf {Y}}}^\intercal {U}_{{\mathbf {Z}},k}= {{\mathbf {X}}}{{\mathbf {Y}}}^\intercal {U}_{{\mathbf {Z}},k}$, we check that $({U}_{{\mathbf {Z}},k},{W})$ is admissible for problem (26). We also check using (24) that $\Vert {{\mathbf {Y}}}- {U}_{{\mathbf {Z}},k}{W}^\intercal {{\mathbf {X}}}\Vert _F^2=\Vert {{\mathbf {Y}}}- {\mathbb {P}}_{{\mathbf {Z}},k} {{\mathbf {Y}}}{\mathbb {P}}_{{{\mathbf {X}}}^\intercal }\Vert _F^2,$ i.e., that $({U}_{{\mathbf {Z}},k},{W})$ reaches the minimum given in (29). In consequence, we have shown that problem (21) and equivalently problem (9) admit the minimizer $A_k^\star ={U}_{{\mathbf {Z}},k}{W}^\intercal = {\mathbb {P}}_{{\mathbf {Z}},k} {{\mathbf {Y}}}{{\mathbf {X}}}^{\dagger }$.

It remains to prove the second part of the theorem, namely the characterization of the approximation error. The sought result follows from standard proper orthogonal decomposition analysis. Indeed, according to (Quarteroni et al. 2015, Proposition 6.1) the first term of the cost function in (28) evaluated at $A_k^\star$ is $\Vert {\mathbf {Z}}- {\mathbb {P}}_{{\mathbf {Z}},k} {\mathbf {Z}}\Vert _F^2= \sum _{i=k+1}^m \sigma _{{\mathbf {Z}},i}^2.$

Proof of Lemma 1

We begin by proving that any minimizer of (21) can be rewritten as $PQ^\intercal$ where $P^\intercal P=I_k$. Indeed, the existence of the SVD of ${\tilde{A}}$ for any minimizer ${\tilde{A}} \in \mathbb {R}^{n \times n }$ guarantees that

$$\begin{aligned} \Vert {{\mathbf {Y}}}- {\tilde{A}} {{\mathbf {X}}}\Vert ^2_F = \Vert {{\mathbf {Y}}}- U_{ {\tilde{A}} }\varSigma _{ {\tilde{A}} }V^\intercal _{ {\tilde{A}} }{{\mathbf {X}}}\Vert ^2_F, \end{aligned}$$

where $U_{ {\tilde{A}} } \in \mathbb {R}^{n \times k}$ possesses orthonormal columns. Making the identification $P=U_{ {\tilde{A}} }$ and $Q=V_{ {\tilde{A}} }\varSigma _{ {\tilde{A}} }$, we verify that $\Vert {{\mathbf {Y}}}- {\tilde{A}} {{\mathbf {X}}}\Vert ^2_F= \Vert {{\mathbf {Y}}}- PQ^\intercal {{\mathbf {X}}}\Vert ^2_F$ and that $P$ possesses orthonormal columns. Next, any solution $PQ^\intercal$ of (21) should satisfy the first-order optimality condition with respect to the jth column denoted $q_j$ of matrix Q, that is

$$\begin{aligned} 2\left[ -{{\mathbf {X}}}{{\mathbf {Y}}}^\intercal p_j + \sum _{i=1}^k (p_i^\intercal p_j) {{\mathbf {X}}}{{\mathbf {X}}}^\intercal q_i\right] =0, \end{aligned}$$

where the jth column of matrix P is denoted $p_j$. In particular, a solution with $P$ possessing orthonormal columns should satisfy ${{\mathbf {X}}}{{\mathbf {Y}}}^\intercal p_j= {{\mathbf {X}}}{{\mathbf {X}}}^\intercal q_j ,$ or in matrix form ${{\mathbf {X}}}{{\mathbf {Y}}}^\intercal P={{\mathbf {X}}}{{\mathbf {X}}}^\intercal Q. \quad$ $\square$

Proof of Proposition 1

We have $A_k^\star = {\mathbb {P}}_{{\mathbf {Z}},k} {{\mathbf {Y}}}{{\mathbf {X}}}^\dagger ={U}_{{\mathbf {Z}},k} {W}^\intercal$ which implies that

$$\begin{aligned} {W}^\intercal {U}_{{\mathbf {Z}},k}={{U}_{{\mathbf {Z}},k}}^\intercal {{\mathbf {Y}}}{{\mathbf {X}}}^\dagger {U}_{{\mathbf {Z}},k}= {{U}_{{\mathbf {Z}},k}}^\intercal {\mathbb {P}}_{{\mathbf {Z}},k} {{\mathbf {Y}}}{{\mathbf {X}}}^\dagger {U}_{{\mathbf {Z}},k}= {{U}_{{\mathbf {Z}},k}}^\intercal {U}_{{\mathbf {Z}},k} {W}^\intercal {U}_{{\mathbf {Z}},k}. \end{aligned}$$

Using the definition of $\zeta _i$’s and $\xi _i$’s in (20), since the $w^r_i$’s and $w^\ell _i$’s are the right and left eigenvectors of ${W}^\intercal {U}_{{\mathbf {Z}},k}$, we verify that

$$\begin{aligned} A^\star _k \zeta _i= {U}_{{\mathbf {Z}},k} {W}^\intercal {U}_{{\mathbf {Z}},k} w^r_i= {U}_{{\mathbf {Z}},k} \lambda _i w^r_i =\lambda _i \zeta _i, \end{aligned}$$

and that

$$\begin{aligned} (A^\star _k)^\intercal \xi _i={\hat{Q}} {{U}_{{\mathbf {Z}},k}}^\intercal {W} w^\ell _i = {W} \lambda _i w^\ell _i=\lambda _i \xi _i. \end{aligned}$$

Finally, $\xi _i^\intercal \zeta _i =1$ is a sufficient condition so that $\xi _i^\intercal A_k^\star \zeta _i =\lambda _i. \quad$ $\square$