Estimating variances in time series kriging using convex optimization and empirical BLUPs

Abstract

We revisit and update estimating variances, fundamental quantities in a time series forecasting approach called kriging, in time series models known as FDSLRMs, whose observations can be described by a linear mixed model (LMM). As a result of applying the convex optimization, we resolved two open problems in FDSLRM research: (1) theoretical existence and equivalence between two standard estimation methods—least squares estimators, non-negative (M)DOOLSE, and maximum likelihood estimators, (RE)MLE, (2) and a practical lack of free available computational implementation for FDSLRM. As for computing (RE)MLE in the case of n observed time series values, we also discovered a new algorithm of order ${\mathcal {O}}(n)$, which at the default precision is $10^7$ times more accurate and $n^2$ times faster than the best current Python(or R)-based computational packages, namely CVXPY, CVXR, nlme, sommer and mixed. The LMM framework led us to the proposal of a two-stage estimation method of variance components based on the empirical (plug-in) best linear unbiased predictions of unobservable random components in FDSLRM. The method, providing non-negative invariant estimators with a simple explicit analytic form and performance comparable with (RE)MLE in the Gaussian case, can be used for any absolutely continuous probability distribution of time series data. We illustrate our results via applications and simulations on three real data sets (electricity consumption, tourism and cyber security), which are easily available, reproducible, sharable and modifiable in the form of interactive Jupyter notebooks.

Appendix

1.1 A.1 Acronyms and abbreviations

See Table 6

Table 6 List of acronyms used in the paper

1.2 A.2 KKT algorithm for NN-(M)DOOLSE

If we consider $\upnu _0 = 0$ which occurs if and only if ${\varvec{e}}= \sum \nolimits _{j=1}^{l}\alpha _j \varvec{v}_j$ in KKT conditions (3.8)–(3.10) then the optimal solution $\tilde{\varvec{\upnu }}$ for NN-(M)DOOLSE is trivial: $\upnu _{0} = 0, \upnu _{j} = \alpha _j^2, j =1, \ldots , l$. For $\upnu _0 \ne 0$ implying $\uplambda _0 =0$, $\upnu _j$ and $\uplambda _j$ cannot be simultaneous zero otherwise it would lead to contradiction between (3.8) and (3.9). Therefore, in the case $\upnu _0 \ne 0$, the complementary slackness condition (3) $\upnu _j\uplambda _j=0,\,\, j\in \{1, \ldots ,l\}$ can be rewritten in the form $\,b_j \upnu _j (1-b_j) \uplambda _j = 0$, where $b_j$ is an auxiliary indicator, which is zero if $\upnu _j = 0, \uplambda _j \ne 0$ and one if $\upnu _j \ne 0, \uplambda _j = 0$.

Using vector $\varvec{b}= (b_1, b_2, \ldots , b_l)' \in \{0,1\}^l$, the derived KKT conditions (2) in Proposition 2 can be described by a $(l+1) \times (l+1)$ matrix function $\mathbf{K }(\varvec{b})= \{K_{ij}(\varvec{b})\}$ and a vector function $\varvec{\upgamma }(\varvec{b}) = (\upgamma _0(\varvec{b}),\upgamma _1(\varvec{b}), \ldots , \upgamma _l(\varvec{b}))'$ as

$$\begin{aligned} \mathbf{K }(\varvec{b})\varvec{\upgamma }(\varvec{b}) = \varvec{q}, \end{aligned}$$

where

$$\begin{aligned} K_{ij}(\varvec{b}) =\left\{ \begin{array}{rl} 0, &{} \text { if } i=0, b_j = 0,\\ -1, &{} \text { if } i=j\ne 0, b_j = 0, \\ G_ {ij}, &{} \text { otherwise.} \end{array} \right. \quad \upgamma _j(\varvec{b}) = \left\{ \begin{array}{cl} \upnu _0, &{} \text { if } j = 0\\ b_j \upnu _j +(1-b_j) \uplambda _j, &{} \text { otherwise.} \end{array} \right. \end{aligned}$$

Applying the Banachiewicz formula, we can write for the inverse of $\mathbf{K }(\varvec{b})$ the following analytic expression

$$\begin{aligned} \mathbf{K}(\varvec{b})^{-1} = \phi ^{-1} \begin{pmatrix} 1 &{} -\varvec{b}'\mathbf{G }_\mathbf{V }\mathbf{D }_{\varvec{b}}^{-1}\\ -\mathbf{D }_{\varvec{b}}^{-1}\mathbf{G }_\mathbf{V }\varvec{j}_{l} &{} \quad \phi \mathbf{D }_{\varvec{b}}^{-1}+\mathbf{D }_{\varvec{b}}^{-1}\mathbf{G }_\mathbf{V }\varvec{j}_{l} \varvec{b}'\mathbf{G }_\mathbf{V }\mathbf{D }_{\varvec{b}}^{-1}\end{pmatrix} \end{aligned}$$

where

$$\begin{aligned} \mathbf{D }_{\varvec{b}}= & {} \text {diag}\left\{ b_j \left\| \varvec{v}_j\right\| ^4+b_j-1\right\} , ~\mathbf{G }_\mathbf{V }= \text {diag}\left\{ \left\| \varvec{v}_j\right\| ^2\right\} , \\ \varvec{j}_{l}= & {} (1, 1, \ldots , 1)' \in \mathbb {R}^{l}, ~\phi = n^{*} -\varvec{b}'\mathbf{G }_\mathbf{V }\mathbf{D }_{\varvec{b}}^{-1}\mathbf{G }_\mathbf{V }\varvec{j}_{l}. \end{aligned}$$

Finally, Proposition 3 guarantees the existence of unique auxiliary vectors ${\tilde{\varvec{\upgamma }}}_{},{\tilde{\varvec{b}}}_{}$:

$$\begin{aligned} {\tilde{\varvec{\upgamma }}}_{}= & {} \{\varvec{\upgamma }; \varvec{\upgamma }=\mathbf{K }(\varvec{b})^{-1}\varvec{q}\ge 0, \varvec{b}\in \{0,1\}^l\} \\ {\tilde{\varvec{b}}}_{}= & {} \{\varvec{b}; {\tilde{\varvec{\upgamma }}}_{}=\mathbf{K }(\varvec{b})^{-1}\varvec{q}, \varvec{b}\in \{0,1\}^l\} \end{aligned}$$

Based on vectors ${\tilde{\varvec{\upgamma }}}_{},{\tilde{\varvec{b}}}_{}$, the NN-(M)DOOLSE $\tilde{\varvec{\upnu }}= ({\tilde{\upnu }}_{0}, {\tilde{\upnu }}_{1}, \ldots , {\tilde{\upnu }}_{l})'$ of $\varvec{\upnu }$ as a solution of KKT conditions has the final form

$$\begin{aligned} {\tilde{\upnu }}_{j} =\left\{ \begin{array}{rl} 0, &{} \text { if } j\ne 0, b_j = 0, \\ {\tilde{\upgamma }}_{j}, &{} \text { otherwise.} \end{array} \right. \end{aligned}$$

Thanks to Proposition 3, it is worth to mention that the given matrix system includes also solutions with $\upnu _0 = 0$.

1.3 A.3 Modern computational technology for FDSLRM kriging

1.3.1 Our computational tools for FDSLRM

As for computational technology, our time series calculations are carried out using free open-source software based on the R statistical language and packages (R Development Core Team 2019; McLeod et al. 2012), the Scientific Python with the SciPy ecosystem (Jones et al. 2001; Oliphant 2007) and free Python-based mathematics software SageMath (Stein and others 2019; Beezer et al. 2013; Zimmermann et al 2018), an open source alternative to the popular commercial computer algebra systems Mathematica or Maple. The simultaneous use of R, SciPy and SageMath provides us valuable cross-checking of our computational results.

Detailed records of our computing with explaining narratives have the form of Jupyter notebooks stored in our free available GitHub repositories devoted to FDSLRM research, https://github.com/fdslrm; see especially a fdslrm repository EBLUP-NE dealing with this paper (Gajdoš et al. 2019c). All notebooks should be seen or studied as static HTML pages via Jupyter nbviewer (https://nbviewer.jupyter.org/, Kluyver et al. (2016)) or interactively as live HTML documents using Binder (https://mybinder.org/, Project Jupyter et al. (2018)) where the code is executable without any need to install or compile the software. To do that, it is only needed to choose and click appropriate links in the Software section of the README homepage in the mentioned repository EBLUP-NE. This easily reproducible way and presentation of our computing with real data were inspired by works (Brieulle et al. 2019; Weiss 2017).

We also started to develop our own R package for FDSLRM, so any potential user can install and use a fully functional version stored in our fdslrm repository R-package (Gajdoš et al. 2019d), which allows the FDLSRM modeling and predictions. Its applications on various real data sets illustrating every step of our three-stage FDSLRM-building approach—formulation (identification), estimation (fitting) and residual diagnostics (checking)—can be found in other fdslrm repository applications (Gajdoš et al. 2019b). All our results obtained from our own implementation were fully consistent with results of standard convex optimization and LMM computational tools which we applied to FDSLRM and are described in the following subsections.

1.3.2 Convex optimization tools for FDSLRM

In nonlinear optimization, there is a variety of highly efficient, fast and reliable open-source and commercial software packages (Cornuéjols et al 2018, chap. 20). Our theoretical consideration and results allow us to apply for estimating $\varvec{\upnu }$ directly the latest, one of the most well-known convex optimization packages CVXPY and CVXR based on disciplined convex programming, written in Python (Diamond and Boyd 2016) and R (Fu et al. 2019).

CVXPY and its R version CVXR are relatively unknown in statistical community. CVXPY is not only a scientific Python library, but also a language with very simple, readable syntax not requiring any expertise in convex optimization and its computer implementation. CVXPY allows the user to specify the mathematical optimization problem naturally following normal mathematical notation as we can see in computing NN-DOOLSE (Fig. 1) where a code easily mimics the non-negative DOOLSE mathematical formulation (3.4) with $\varvec{{\Sigma }}_{\varvec{\upnu }}= \sigma _0^2\mathbf{I }_n+\mathbf{V }\mathbf{D }_{\varvec{\upnu }}\mathbf{V }'=\upnu _0\mathbf{I }_n+\mathbf{V }\text {diag}\left\{ \upnu _j\right\} \mathbf{V }'$.

Fig. 1

CVXPY code for computing DOOLSE in Jupyter environment

CVXPY implements not only convex optimization solvers using interior-point numerical methods which are extremely reliable and fast, but also verifies convexity of the given problem using rules of disciplined convex programming (Grant et al. 2006). CVXPY was inspired by MATLAB optimization package CVX (Grant and Boyd 2018) still used in many references, e.g. Cornuéjols et al. (2018). However, in CVXPY (CVXR) the user can easily combine convex optimization and simplicity of Python (R) language together with its high-level features such as object-oriented design or parallelism.

It is worth to mention that unlike our KKT algorithm, both CVXPY and CVXR are able to solve non-negative DOOLSE and MDOOLSE in a general FDSLRM without the orthogonality condition. In FDSLRM, interior-points numerical methods have complexity $n^3$ for one iteration or $\log (1/\varepsilon )$ times bigger for the complete computation with a precision $\varepsilon$ of the required optimal solution (Boyd and Vandenberghe 2009).

1.3.3 LMM computational tools for FDSLRM

In our previous paper (Gajdoš et al. 2017), we stated that no current package in R is directly and effectively suitable for FDSLRM. Thanks to a detailed study of (Demidenko 2013; Galecki and Burzykowski 2013) we were successfully directed and instructed how to implement the FDSLRM variance structure into one of the best R packages for LMM known as nlme (Pinheiro and Bates 2009; Pinheiro et al 2018). After that, inspired by nlme, we found another R package called sommer, also suitable for FDSLRM fitting (Covarrubias-Pazaran 2016; Cornuéjols et al. 2018).

Simultaneously, thanks to online computing environment CoCalc for SageMath (Stein and others 2019) with the possibility to run computations and codes for free in many other programming languages and open softwares, we are also able to run and test MATLAB function mixed (Witkovský 2018, 2012) primarily intended for estimating variances in LMMs. On the basis of successful tests, we wrote its R version called MMEinR, which is also available in our GitHub Repository (Gajdoš 2019).

The mentioned LMM packages can also handle (RE)MLE in the general FDSLRM without orthogonality or full-rankness restriction. The packages use iterative methods based on EM algorithm or Henderson MME together with some version of the Newton-Raphson method whose complexity is generally at least $n^3$.

We are fully aware that another efficient implementation for estimating variances in LMMs provides lme4 package (Bates et al. 2015) or SAS package PROC MIXED (Stroup et al. 2018; SAS Institute Inc. 2018). As for lme4, which is much faster than nlme, we found that the package does not allow implementing FDSLRM covariance structure using lme4 standard input procedure. As SAS laymans, we also tried a university edition of SAS, free from 2014, but we left it after running into initial problems with Windows installation process in a virtual machine environment and subsequently with a more sophisticated programming language than Python or R. Therefore our knowledge with SAS remained only at the theoretical level.

Finally, we point out that thanks to SageMath and our very fast KKT algorithm, we were able to compute (in real time) NN-(M)-DOOLSE in the toy example with infinite precision—as the exact closed-form (algebraic) numbers. To get an explicit idea, e.g. for our toy model, we got these results for NN-MDOOLSE $\tilde{\varvec{\upnu }}$ identical with REMLE

$$\begin{aligned} \tilde{\varvec{\upnu }} = \begin{pmatrix} \tilde{\sigma }_0^2 \\ \tilde{\sigma }_1^2 \\ \tilde{\sigma }_2^2 \\ \tilde{\sigma }_3^2 \\ \sigma _4^2 \end{pmatrix} = \begin{pmatrix} -\frac{6569}{4080} \, \sqrt{3} \sqrt{2} - \frac{13207}{4080} \, \sqrt{3} - \frac{22533}{6800} \, \sqrt{2} + \frac{312727}{20400} \\ \frac{6569}{48960} \, \sqrt{3} \sqrt{2} + \frac{275509}{244800} \, \sqrt{3} + \frac{7511}{27200} \, \sqrt{2} + \frac{50461}{244800} \\ \frac{6569}{48960} \, \sqrt{3} \sqrt{2} + \frac{37687}{48960} \, \sqrt{3} + \frac{7511}{27200} \, \sqrt{2} - \frac{93427}{244800} \\ \frac{6569}{48960} \, \sqrt{3} \sqrt{2} + \frac{13207}{48960} \, \sqrt{3} + \frac{11081}{27200} \, \sqrt{2} - \frac{66779}{61200} \\ \frac{6569}{48960} \, \sqrt{3} \sqrt{2} + \frac{13207}{48960} \, \sqrt{3} + \frac{43637}{48960} \, \sqrt{2} - \frac{17377}{61200} \end{pmatrix} \approx \begin{pmatrix} 1.093 \\ 2.875 \\ 1.671 \\ 0.281 \\ 1.772 \end{pmatrix}. \end{aligned}$$

It means that we can compute real numerical errors in results for all used computational tools to explore their quality from the viewpoint of numerical precision.

A.4 The BLUP-NE method — proofs

Existence of inversions for $\mathbf{U} _{\varvec{\upnu }}$,  $\mathbf{H} _{\varvec{\upnu }}$

To prove nonsingularity of $\mathbf{U} _{\varvec{\upnu }}$, $\mathbf{H} _{\varvec{\upnu }}$, it is sufficient to show that both matrices have non-zero determinants. Using idempotence of $\mathbf{M} _\mathbf{F }$ and determinant expression (3.17), we can write for determinants of $\mathbf{U} _{\varvec{\upnu }}$, $\mathbf{H} _{\varvec{\upnu }}$

$$\begin{aligned} \det \mathbf{H} _{\varvec{\upnu }}= & {} \det (\sigma _0^2 \mathbf{I }_l+\mathbf{V }'\mathbf{V }\mathbf{D }_{\varvec{\upnu }}) = (-\sigma _0^2)^{l-n}\det (\sigma _0^2 \mathbf{I }_l+\mathbf{V }\mathbf{D }_{\varvec{\upnu }}\mathbf{V }'), \\ \det \mathbf{U} _{\varvec{\upnu }}= & {} \det (\sigma _0^2 \mathbf{I }_l+\mathbf{V }'\mathbf{M} _\mathbf{F }\mathbf{M} _\mathbf{F }\mathbf{V }\mathbf{D }_{\varvec{\upnu }}) = (-\sigma _0^2)^{l-n}\det (\sigma _0^2 \mathbf{I }_l+\mathbf{M} _\mathbf{F }\mathbf{V }\mathbf{D }_{\varvec{\upnu }}\mathbf{V }'\mathbf{M} _\mathbf{F }). \end{aligned}$$

Now we can see that the sum of positive definite matrix $\sigma _0^2\mathbf{I }_l\,\, (\sigma _0^2>0)$ and each of positive semidefinite matrices $\mathbf{V }\mathbf{D }_{\varvec{\upnu }}\mathbf{V }', \,\,\mathbf{M} _\mathbf{F }\mathbf{V }\mathbf{D }_{\varvec{\upnu }}\mathbf{V }'\mathbf{M} _\mathbf{F }$ is a positive definite matrix, whose determinant is always positive.

Proof (Theorem 2)

Employing Lemma 1 and the well-known standard expressions for mean values and covariances of invariant quadratic estimators [see e.g. Christensen (2011), Puntanen et al. (2013)] $E_{\varvec{\upnu }}\left\{ \varvec{X}'\mathbf{A }\varvec{X}\right\} =\text {tr}(\mathbf{A }\varvec{{\Sigma }}_{\varvec{\upnu }})$ and if $\varvec{X}\sim {\mathcal {N}}_n(\mathbf{F }\varvec{\beta }, \varvec{{\Sigma }}_{\varvec{\upnu }})$ then ${Cov}_{\varvec{\upnu }}\left\{ \varvec{X}'\mathbf{A }\varvec{X}, \varvec{X}'\mathbf{B }\varvec{X}\right\} =2\,\text {tr}(\mathbf{A }\varvec{{\Sigma }}_{\varvec{\upnu }}\mathbf{B }\varvec{{\Sigma }}_{\varvec{\upnu }})$, we have

• (1)
  

  According to (3) of Lema 1$(\mathbf{T }_{\varvec{\upnu }}\varvec{{\Sigma }}_{\varvec{\upnu }}\mathbf{T }_{\varvec{\upnu }}')_{jj}= \mathbf{D }_{\varvec{\upnu }jj}-\sigma ^2(\mathbf{W} _{\varvec{\upnu }}^{-1})_{jj}= \sigma ^2_j-\sigma ^2(\mathbf{W} _{\varvec{\upnu }}^{-1})_{jj}$.

• (2)
  

  As a consequence of (1) $D_{\varvec{\upnu }}\{\mathring{{\sigma }}_{j}^2\} =2(\sigma ^2_j-\sigma ^2(\mathbf{W} _{\varvec{\upnu }}^{-1})_{jj})^ 2$.

• (3) In a similar way as in (2)

  $$\begin{aligned} {Cov}_{\varvec{\upnu }}\left\{ \mathring{{\sigma }}_{i}^2,\mathring{{\sigma }}_{j}^2\right\}= & {} 2\text {tr}(\varvec{\tau }_{ i}\varvec{\tau }_{ i}'\varvec{{\Sigma }}_{\varvec{\upnu }}\varvec{\tau }_{ j}\varvec{\tau }_{ j}' \varvec{{\Sigma }}_{\varvec{\upnu }})= 2\text {tr}(\varvec{\tau }_{ i}'\varvec{{\Sigma }}_{\varvec{\upnu }}\varvec{\tau }_{ j}\varvec{\tau }_{ j}' \varvec{{\Sigma }}_{\varvec{\upnu }}\varvec{\tau }_{ i})= 2(\varvec{\tau }_{ i}'\varvec{{\Sigma }}_{\varvec{\upnu }}\varvec{\tau }_{ j})^2 \\= & {} 2(\mathbf{T }_{\varvec{\upnu }} \varvec{{\Sigma }}_{\varvec{\upnu }}\mathbf{T }_{\varvec{\upnu }}')^2_{ij}= 2(0-\sigma ^2 (\mathbf{W} _{\varvec{\upnu }}^{-1})_{ij})^2= 2(\sigma ^2 (\mathbf{W} _{\varvec{\upnu }}^{-1})_{ij})^2. \end{aligned}$$
• (4)
  

$\square$