161-A3.1

• Activity: Mutual Information and Channel Capacity
• Instructions: In this activity, you are tasked to
  • Walk through the examples.
  • Calculate the channel capacity of different channel models.
• Should you have any questions, clarifications, or issues, please contact your instructor as soon as possible.

Example 1: Mutual Information

Given the following probabilities:

$X$: Blood Type, $Y$: Chance for Skin Cancer

	A	B	AB	O
Very Low	1/8	1/16	1/32	1/32
Low	1/16	1/8	1/32	1/32
Medium	1/16	1/16	1/16	1/16
High	1/4	0	0	0

To get the entropies of $X$ and $Y$, we need to calculate the marginal probabilities:

$P_{X}=\{{\tfrac {1}{2}},{\tfrac {1}{4}},{\tfrac {1}{8}},{\tfrac {1}{8}}\}$

(1)

$P_{Y}=\{{\tfrac {1}{4}},{\tfrac {1}{4}},{\tfrac {1}{4}},{\tfrac {1}{4}}\}$

(2)

And since:

$H\left(A\right)=\sum _{i=1}^{n}P\left(a_{i}\right)\cdot \log _{2}\left({\frac {1}{P\left(a_{i}\right)}}\right)$

(3)

We get:

$H\left(X\right)={\frac {1}{2}}\log _{2}2+{\frac {1}{4}}\log _{2}4+{\frac {1}{8}}\log _{2}8+{\frac {1}{8}}\log _{2}8={\frac {7}{4}}\,\mathrm {bits}$

(4)

$H\left(Y\right)={\frac {1}{4}}\log _{2}4+{\frac {1}{4}}\log _{2}4+{\frac {1}{4}}\log _{2}4+{\frac {1}{4}}\log _{2}4=2\,\mathrm {bits}$

(5)

Calculating the conditional entropies using:

$H\left(X\mid Y\right)=\sum _{i=1}^{4}\sum _{j=1}^{4}P\left(x_{i},y_{j}\right)\cdot \log _{2}\left({\frac {P\left(y_{j}\right)}{P\left(x_{i},y_{j}\right)}}\right)$

(6)

We get:

${\begin{aligned}H\left(X\mid Y\right)=&{\frac {1}{8}}\log _{2}{\frac {1/4}{1/8}}+{\frac {1}{16}}\log _{2}{\frac {1/4}{1/16}}+{\frac {1}{16}}\log _{2}{\frac {1/4}{1/16}}+{\frac {1}{4}}\log _{2}{\frac {1/4}{1/4}}\\&+{\frac {1}{16}}\log _{2}{\frac {1/4}{1/16}}+{\frac {1}{8}}\log _{2}{\frac {1/4}{1/8}}+{\frac {1}{16}}\log _{2}{\frac {1/4}{1/16}}+0\log _{2}{\frac {1/4}{0}}\\&+{\frac {1}{32}}\log _{2}{\frac {1/4}{1/32}}+{\frac {1}{32}}\log _{2}{\frac {1/4}{1/32}}+{\frac {1}{16}}\log _{2}{\frac {1/4}{1/16}}+0\log _{2}{\frac {1/4}{0}}\\&+{\frac {1}{32}}\log _{2}{\frac {1/4}{1/32}}+{\frac {1}{32}}\log _{2}{\frac {1/4}{1/32}}+{\frac {1}{16}}\log _{2}{\frac {1/4}{1/16}}+0\log _{2}{\frac {1/4}{0}}\\=&1\end{aligned}}$

(7)

Sources

• Yao Xie's slides on Entropy and Mutual Information