Covariance Matrix Formula

Covariance is a term significantly used in a particular branch of mathematics such as statistics and probability theory. Many articles and reading elements on probabilistic theories and statistics presume a basic understanding of terms such as mean, standard deviation, variance, correlation, sample sizes, population and covariance. Let us understand the term covariance and how it is related to matrices in the following article. 

The aim of the article is to define what is covariance matrix, along with the concepts related to the covariance matrix formula such as its properties, and calculate covariance matrix. The covariance matrix formula will be helpful for the readers in understanding how to calculate covariance matrix. Let’s start learning!!

Covariance Matrix

We know that the term covariance is used to measure the association (or relationship) and the dependency between any two given variables.  Generally, readers often found it difficult to recognize the difference between the terms covariance and the term Correlation.

Both the terms covariance and correlation are used to measure the relationship between any two data points. The term Covariance symbolises the direction of the linear relationship between variables.  At the same time, the term correlation measures both the strength and direction of the linear relationship between two variables. In fact, it can be said that correlation is a function of covariance. The major difference between the covariance and the correlation is the fact that correlation values are always standardized whereas, covariance values variables or they are not standardized. 

We can get the coefficient of correlation between the two variables by dividing the covariance of these variables by the outcome of the standard deviations of the same values. If we recall the definition of Standard Deviation, we know that it typically measures the absolute variability of the distribution of the data points. When we divide the covariance values by the standard deviation, it basically scales the value down to a confined range (or limited range) of -1 to +1. This is specifically the range of the correlation values.

Calculate Covariance Matrix

Now, let us understand the covariance matrix formula and how to calculate the covariance matrix. In probabilistic theory and statistics, a covariance matrix is basically a square matrix giving the covariance relation between the set of pairs of elements of a considered random vector. All the covariance matrices are always symmetric in nature and also positive semi-definite and its chief diagonal includes variances i.e., the covariance of every element with itself. The covariance matrix is also known as the autocovariance matrix, dispersion matrix, variance matrix, or variance covariance matrix.

Therefore, the covariance matrix basically just generalizes the notion of variance to multiple dimensions and it is the key concept to be learnt while learning the linear vector space. To be more precise, let us consider an example, the variation in a collection of random data points in two-dimensional space can not be described fully by a single number, nor would the variances in the x and y directions include all of the required basic information. A \[2\times 2\] would be required to fully describe the two-dimensional variation.

The covariance of two variables (x and y) can be represented as cov (x,y). If \[E[x]\] is the expected value or mean of a sample ‘x’, then cov (x , y) can be represented in the following way:

\[\Rightarrow Cov (x, y) = E[(x-\mu_{x})(y-\mu_{y}) ]\]

\[\Rightarrow Cov (x, y) = E[xy] - E(x) E(y) : \forall \mu_{x}\] and \[ \mu_{y} = E(x)\] and \[E(y) \]

\[\Rightarrow Cov(x,y) = E[xy] - \mu_{x} \mu_{y} \]

Now, let us define the covariance matrix and its formula as follows. Before we start with the covariance matrix, let us first define a mean vector. The mean vector includes the means of every variable and the covariance matrix includes the variances of the variables along the principal diagonal and the covariances among all pairs of variables in the different matrix positions.

The covariance formula for computing the covariance of the variables x and y is given by:

\[\Rightarrow Cov (x, y) = \frac{ \displaystyle\sum\limits_{i=1}^n (X_{i} - \bar{x})(Y_{i} - \bar{y})}{n-1} \]

Where,

\[\bar{x}\] - The mean of the \[x\]

\[\bar{y}\] - The mean of the \[y\]

The mean vector is often regarded as the centroid and the variance-covariance matrix is known as the dispersion or dispersion matrix. In few writings we find that it is mentioned as variance and covariance matrix, we should remember an important and essential point which says that both the terms variance-covariance matrix and covariance matrix are utilised reciprocally. 

If the covariance matrix is positive definite, then the distribution of X is non-degenerate; otherwise, it is degenerate. For the random vector, x the covariance matrix plays the same role as the variance of a random variable. If the variances of the random variables \[x_{1}...x_{k} \] are all equal to 1, then the covariance matrix of \[ x = x_{1}...x_{k} \] is the same as the correlation matrix.

The sample covariance matrix for the sample \[x_{1}...x_{m} \], where the \[x_{m}, m = 1….n\], are independent and identically-distributed random k- dimensional vectors, consists of the variance and covariance estimators:

\[\Rightarrow S = \frac{1}{n-1} \displaystyle\sum\limits_{m=1}^m (x_{m} - \bar{x})(x_{m} - \bar{x})^{T}  \]

Where the vector \[\bar{x}\] is the arithmetic mean of the \[x_{1}....x_{n}\]. If the \[x_{1}....x_{n}\] are multivariate normally distributed with a covariance matrix \[\sum,\] then \[\frac{s(n-1)}{n} \] is the maximum-likelihood estimator of \[\sum;\] in this case the joint distribution of the elements of the matrix \[S(n-1)\] is known as the Wishart distribution. And it is one of the fundamental distributions in multivariate statistical analysis by means of which hypotheses concerning the covariance matrix \[\sum\] can be tested.

Example:

Let us have a look at the covariance matrix example:

1). Calculation of Covariance Matrix from the Data Matrix provided.

Sol:

Suppose the data matrix \[y_{1} = 5_{z_{1}-z_{2}}\] and \[y_{1} = 2_{z_{2}}\] with rows corresponding to subjects and columns are variables. Calculate a mean for each variable and replace the data matrix.

Now the matrix of deviations from the mean is: \[y - \bar{y}\]

Therefore the covariance matrix of the observation is:

\[ Z = \begin{pmatrix}-2 & -4 \\-1 & 2 \\0 &0 \\1 &-2 \\ 2&4 \end{pmatrix}\]

The diagonal elements of this matrix are the variances of the variables, and the off-diagonal elements are the covariances between the variables.

\[ \frac{1}{N-1} Z^{1}Z = \frac{1}{4} \begin{pmatrix}2 & -1& 0& 1&2 \\-4 & 2&0&-2&4 \end{pmatrix} \begin{pmatrix}-2 & -4 \\-1 & 2 \\0 &0 \\1 &-2 \\ 2&4 \end{pmatrix} \]

\[ = \frac{1}{4} \begin{pmatrix}10 & 12 \\12 & 40 \end{pmatrix}\]

\[ = \begin{pmatrix}2.5 & 3.0 \\3.0 & 10.0 \end{pmatrix}\]

\[ = \begin{pmatrix}S_{x}^{2} & S_{xy} \\S_{xy} & S^{2}_{x} \end{pmatrix}\]