161-A1.1

Let's look at a few applications of the concept of information and entropy.

Surprise! The Unexpected Observation

Information can be thought of as the amount of surprise at seeing an event. Note that a highly probable outcome is not surprising. Consider the following events:

Event	Probability	Information (Surprise)
Someone tells you $1=1$.	$1$	$\log _{2}\left(1\right)=0$
You got the wrong answer on a 4-choice multiple choice question.	${\frac {3}{4}}$	$\log _{2}\left({\frac {4}{3}}\right)=0.415\,\mathrm {bits}$
You guessed correctly on a 4-choice multiple choice question.	${\frac {1}{4}}$	$\log _{2}\left(4\right)=2\,\mathrm {bits}$
You got the correct answer in a True or False question.	${\frac {1}{2}}$	$\log _{2}\left(2\right)=1\,\mathrm {bit}$

Student Grading

How much information can we get from a single grade? Note that the maximum entropy occurs when all the grades have equal probability.

• For Pass/Fail grades, the possible outcomes are: $\{\mathrm {P} ,\mathrm {F} \}$ with probabilities $\{{\tfrac {1}{2}},{\tfrac {1}{2}}\}$. Thus,

$H\left(P\right)=\sum _{i=1}^{n}p_{i}\cdot \log _{2}\left({\frac {1}{p_{i}}}\right)={\frac {1}{2}}\cdot \log _{2}\left(2\right)+{\frac {1}{2}}\cdot \log _{2}\left(2\right)=1\,\mathrm {bit}$

(1)

• For grades = $\{1.00,2.00,3.00,4.00,5.00\}$ with probabilities $\{{\tfrac {1}{5}},{\tfrac {1}{5}},{\tfrac {1}{5}},{\tfrac {1}{5}},{\tfrac {1}{5}}\}$, we get:

$H\left(P\right)=\sum _{i=1}^{n}p_{i}\cdot \log _{2}\left({\frac {1}{p_{i}}}\right)=5\cdot {\frac {1}{5}}\cdot \log _{2}\left(5\right)=2.32\,\mathrm {bits}$

(2)

• For grades = $\{1.00,1.50,2.00,2.50,3.00,4.00,5.00\}$ with probabilities $\{{\tfrac {1}{7}},{\tfrac {1}{7}},{\tfrac {1}{7}},{\tfrac {1}{7}},{\tfrac {1}{7}},{\tfrac {1}{7}},{\tfrac {1}{7}}\}$, we have:

$H\left(P\right)=\sum _{i=1}^{n}p_{i}\cdot \log _{2}\left({\frac {1}{p_{i}}}\right)=7\cdot {\frac {1}{7}}\cdot \log _{2}\left(7\right)=2.81\,\mathrm {bits}$

(3)

• If we have all the possible grades $\{1.00,1.25,1.50,1.75,2.00,2.25,2.50,2.75,3.00,4.00,5.00,\mathrm {INC} ,\mathrm {DRP} ,\mathrm {LOA} \}$ with probabilities $\{{\tfrac {1}{14}},{\tfrac {1}{14}},{\tfrac {1}{14}},{\tfrac {1}{14}},{\tfrac {1}{14}},{\tfrac {1}{14}},{\tfrac {1}{14}},{\tfrac {1}{14}},{\tfrac {1}{14}},{\tfrac {1}{14}},{\tfrac {1}{14}},{\tfrac {1}{14}},{\tfrac {1}{14}},{\tfrac {1}{14}}\}$, we have:

$H\left(P\right)=\sum _{i=1}^{n}p_{i}\cdot \log _{2}\left({\frac {1}{p_{i}}}\right)=14\cdot {\frac {1}{14}}\cdot \log _{2}\left(14\right)=3.81\,\mathrm {bits}$

(4)