Covariance Matrix

Definition

It is convenient to define a covariance matrix by using multi-variate random variables (mrv): X = (X ₁, …, X _d ) ^⊤ ​. For univariate random variables X _i and X _j , their covariance is defined as:

$$\textrm{ Cov}({X}_{i},{X}_{j}) = \mathbb{E}\left [({X}_{i} - {\mu }_{i})({X}_{j} - {\mu }_{j})\right ],$$

where μ _i is the mean of X _i : ${\mu }_{i} = \mathbb{E}[{X}_{i}]$. As a special case, when i = j, then we get the variance of X _i , Var(X _i ) = Cov(X _i , X _i ). Now in the setting of mrv, assuming that each component random variable X _i has finite variance under its marginal distribution, the covariance matrix Cov(X, X) can be defined as a d-by-d matrix whose (i, j)-th entry is the covariance:

$${(\textrm{ Cov}(\mathbf{X},\mathbf{X}))}_{ij} = \textrm{ Cov}({X}_{i},{X}_{j}) = \mathbb{E}\left [({X}_{i} - {\mu }_{i})({X}_{j} - {\mu }_{j})\right ].$$

And its inverse is also called precision matrix.

Motivation and Background

The covariance between two univariate random variables...