Joint entropy, conditional entropy, and mutual information

Joint Entropies

In this module, we'll discuss several extensions of entropy. Let's begin with joint entropy. Suppose we have a random variable $X$ with elements $\{x_{1},x_{2},...,x_{i},...,x_{n}\}$ and random variable $Y$ with elements $\{y_{1},y_{2},...,y_{j},...,y_{m}\}$. We define the joint entropy of $H(X,Y)$ as:

$H(X,Y)=-\sum _{i=1}^{n}\sum _{j=1}^{m}P(x_{i},y_{j})\log \left(P(x_{i},y_{j})\right)$

(1)

Take note that $H(X,Y)=H(Y,X)$. This is trivial. From our previous discussion about the decision trees of a game, we used joint entropies to calculate the overall entropy.

Conditional Entropies

There are two steps to understand conditional entropies. The first is the uncertainty of a random variable changes when a single outcome occurs. Suppose we have the same random variables $X$ and $Y$ defined earlier in joint entropies. Let's denote $P(y_{j}|x_{i})$ as the conditional probability of $y_{j}$ when $x_{i}$ event happened. We define the entropy $H(Y|x_{i})$ as the entropy of the random variable $Y$ given a single outcome $x_{i}$ happened. Mathematically this is:

$H(Y|x_{i})=--\sum _{j=1}^{m}P(y_{j}|x_{i})\log \left(P(y_{j}|x_{i})\right)$

(2)

Stated in a different way, $H(Y|x_{i})$ is our measure of uncertainty about $Y$ when we know $x_{i}$ occurred. Take note that equation 2 just pertains to the uncertainty when a single event happened. We'll extend this to what would the uncertainty be when any of the events in $X$ happens.