161-A3.1

Activity: Mutual Information and Channel Capacity
Instructions: In this activity, you are tasked to
- Walk through the examples.
- Calculate the channel capacity of different channel models.
Should you have any questions, clarifications, or issues, please contact your instructor as soon as possible.

Example 1: Mutual Information

Given the following probabilities:

To get the entropies of $X$ and $Y$ , we need to calculate the marginal probabilities:

P_{X}=\{{\tfrac {1}{2}},{\tfrac {1}{4}},{\tfrac {1}{8}},{\tfrac {1}{8}}\}

(1)

P_{Y}=\{{\tfrac {1}{4}},{\tfrac {1}{4}},{\tfrac {1}{4}},{\tfrac {1}{4}}\}

(2)

And since:

H\left(A\right)=\sum _{i=1}^{n}P\left(a_{i}\right)\cdot \log _{2}\left({\frac {1}{P\left(a_{i}\right)}}\right)

(3)

We get:

H\left(X\right)={\frac {1}{2}}\log _{2}2+{\frac {1}{4}}\log _{2}4+{\frac {1}{8}}\log _{2}8+{\frac {1}{8}}\log _{2}8={\frac {7}{4}}\,\mathrm {bits} =1.75\,\mathrm {bits}

(4)

H\left(Y\right)={\frac {1}{4}}\log _{2}4+{\frac {1}{4}}\log _{2}4+{\frac {1}{4}}\log _{2}4+{\frac {1}{4}}\log _{2}4=2\,\mathrm {bits}

(5)

Calculating the conditional entropies using:

H\left(X\mid Y\right)=\sum _{i=1}^{4}\sum _{j=1}^{4}P\left(x_{i},y_{j}\right)\cdot \log _{2}\left({\frac {P\left(y_{j}\right)}{P\left(x_{i},y_{j}\right)}}\right)={\frac {11}{8}}\,\mathrm {bits} =1.375\,\mathrm {bits}

(6)

H\left(Y\mid X\right)=\sum _{i=1}^{4}\sum _{j=1}^{4}P\left(x_{i},y_{j}\right)\cdot \log _{2}\left({\frac {P\left(x_{i}\right)}{P\left(x_{i},y_{j}\right)}}\right)={\frac {13}{8}}\,\mathrm {bits} =1.625\,\mathrm {bits}

(7)

Note that $H\left(X\mid Y\right)\neq H\left(Y\mid X\right)$ . Calculating the mutual information, we get:

I\left(A;B\right)=H\left(X\right)-H\left(X\mid Y\right)={\frac {7}{4}}-{\frac {11}{8}}=0.375\,\mathrm {bits}

(8)

Or equivalently:

I\left(A;B\right)=H\left(Y\right)-H\left(Y\mid X\right)=2-{\frac {13}{8}}=0.375\,\mathrm {bits}