# Correlation and Mutual Information

Peter Krafft · February 13, 2013

Mutual information is a quantification of the dependency between random variables. It is sometimes contrasted with linear correlation since mutual information captures nonlinear dependence. In this short note I will discuss the relationship between these quantities in the case of a bivariate Gaussian distribution, and I will explore two implications of that relationship.

As shown below, in the case of $X$ and $Y$ having a bivariate Normal distribution, mutual information is a monotonic transformation of correlation,

$$

I(X,Y) = -\frac{1}{2} \ln

\left(1 -

Corr(X,Y)^2

\right).

$$

This relationship has a couple implications that I would like to highlight. First, it proves that lack of correlation in the bivariate Normal distribution implies independence, i.e. when the correlation between $X$ and $Y$ is zero, the mutual information will also be zero. More interestingly, if you are willing to assume that the marginal distributions of $X$ and $Y$ are Normal but you are not willing to assume joint normality, this result provides a lower bound on the mutual information between $X$ and $Y$. This lower bound follows from the maximum entropy property of the Normal distribution.

Formally, we have that the entropy of a univariate Gaussian random variable, $X$,

$$H(X) = \frac{1}{2} \ln (2\pi e \sigma_x^2).$$

For a bivariate Gaussian random variable, $(X,Y)$,

$$H(X,Y) = \frac{1}{2} \ln ((2\pi e)^2 (\sigma_x^2 \sigma_y^2 - \sigma_{xy}^2)),$$

where $\sigma_{xy}$ is the covariance of $X$ and $Y$.

Then the mutual information,

\begin{align*}

I(X,Y) &= H(X) + H(Y) - H(X,Y)

\\

&= \frac{1}{2} \ln

\left(

\frac{\sigma_x^2\sigma_y^2}{\sigma_x^2 \sigma_y^2 - \sigma_{xy}^2}

\right)

\\

&= \frac{1}{2} \ln

\left(

\frac{\sigma_y^2}{\sigma_y^2 - \sigma_{xy}^2/\sigma_x^2 }

\right)

\\

&= -\frac{1}{2} \ln

\left(

\frac{\sigma_y^2 - \sigma_{xy}^2/\sigma_x^2 }{\sigma_y^2}

\right)

\\

&= -\frac{1}{2} \ln

\left(1 -

\left(\frac{\sigma_{xy}}{\sigma_x \sigma_y}\right)^2

\right)

\\

&= -\frac{1}{2} \ln

\left(1 -

Corr(X,Y)^2

\right).

\end{align*}

The lower bound follows from the following argument. Consider two other random variables, $X'$ and $Y'$, that have the same covariance as $X$ and $Y$ but are jointly normally distributed. Note that since we have assumed that the marginal distributions of $X$ and $Y$ are Normal, we have $H(X) = H(X')$ and $H(Y) = H(Y')$, and by the maximum entropy property of the Normal distribution, we have $H(X,Y) \leq H(X',Y')$. The result is then straightforward:

\begin{align*}

I(X,Y) &= H(X) + H(Y) - H(X,Y)

\\

&\geq

H(X') + H(Y') - H(X',Y')

\\

&= -\frac{1}{2} \ln

\left(1 -

Corr(X',Y')^2

\right)

\\

&= -\frac{1}{2} \ln

\left(1 -

Corr(X,Y)^2

\right).

\end{align*}

Thanks are due to Vikash Mansinghka for suggesting the first part of this exercise.