The Optimal Regularized Weighted Least-Squares Method for Impulse Response Estimation

Abstract

The system identification literature has been going through a recent paradigm change with the emergent use of regularization and kernel-based methodologies to identify the process’ impulse response. However, the literature is quite scarce when dealing with processes that possess colored additive output noise. In this case, the current alternative is to identify a system predictor instead, which can be somewhat unfavorable in situations where the process’ model is strictly necessary. So, the main objective of this paper is to introduce a novel regularized system identification methodology that has been specifically developed for the colored output noise scenario. Such methodology is based on the Bayesian perspective of the identification procedure, and it results in the regularized weighted least-squares method, which can be interpreted as an extension of the well-known regularized least squares. The paper also presents the method’s statistical properties, optimal choices, and parametrization structures for both the regularization and weighting matrices, along with a dedicated algorithm to estimate these matrices. Finally, Monte Carlo simulations are performed to demonstrate the method’s efficiency.

Appendix A Matrix Calculus Results

Here we present some matrix calculus results that were used to calculate the results in Sect. 3 regarding the MSE trace optimization for the RWLS estimator.

1.1 A.1 Derivatives with Relation to P

Let us start with some generic derivatives that were used to compute the derivative of the MSE trace with relation to P:

$$\begin{aligned}&tr\left[ \dfrac{\partial \left( PA+I\right) ^{-1}B}{\partial P_{ij}}\right] = -\left[ ZB^TZA^T\right] _{ij}, \end{aligned}$$

(A1)

$$\begin{aligned}&tr\left[ \dfrac{A\partial \left( PBP^T\right) C}{\partial P_{ij}}\right] = \big [BP^TCA\big ]_{ji} +\big [CAPB\big ]_{ij}, \end{aligned}$$

(A2)

$$\begin{aligned}&tr\left[ \dfrac{A \partial \left( BP^T+I\right) ^{-1}}{\partial P_{ij}}\right] = -\left[ WAWB\right] _{ij}, \end{aligned}$$

(A3)

where $Z=\left( A^TP^T+I\right) ^{-1}$, and $W=\left( BP^T+I\right) ^{-1}$.

1.2 A.2 Derivatives with Relation to M

Now, some results regarding the derivatives of the MSE trace with relation to M:

$$\begin{aligned}&tr\left[ \dfrac{\partial \left( AMB+I\right) ^{-1}C}{\partial M_{ij}}\right] = -\left[ A^TVC^TVB^T\right] _{ij}, \end{aligned}$$

(A4)

$$\begin{aligned}&tr\left[ \dfrac{\partial A\left( BM\Sigma M^TC\right) D}{\partial M_{ij}}\right] = \big [CDABM\Sigma \big ]_{ij}+ \big [\Sigma M^TCDAB\big ]_{ji}, \end{aligned}$$

(A5)

$$\begin{aligned}&tr\left[ \dfrac{\partial A\left( BM^TC+I\right) ^{-1}}{\partial M_{ij}}\right] = - \left[ CUAUB\right] _{ij}. \end{aligned}$$

(A6)

with $V=\left( B^TM^TA^T+I\right) ^{-1}$, and $U=\left( BM^TC+I\right) ^{-1}$.