2S2122 Activity 2.2

Instructions

There are two parts to this exercise.

1. Read through the discussion about information and language.

2. After understanding the discussion, we'll replicate it so that you can get hands on experience on the analysis.

Submission guidelines:

• For every programming exercise, you are to submit your .ipynb file into your respective submission bin.
• To download your .ipynb file, first in your Google Colab page go to File > Download > Download .ipynb file.
• Don't forget to rename your .ipynb file with the following format "class_lastname_firstname_studentnumber.ipynb".

Information and Language

One interesting application of recognizing how entropy plays a role in our everyday life is to determine the average information per letter in a language. In this programming exercise, we will explore and try to understand what entropy means for the English language. Before we begin let's set the context of our problem first.

• We'll be characterizing the information content of the English language based only on the frequency of letters.
• We'll expand this problem to a block of letters but get the relative entropy for a single letter only.
• We'll exclude the conditional probabilities for this exercise because that will be more challenging to build.

Let's first begin by defining the word N-block as the number of letters we are combining to represent a single outcome in a random variable $X$. Consider the sentence:

"The quick brown fox jumps over the lazy dog"

When we say we want to get the 1-block, then $X$ contains $\{t,h,e,q,u,i,c,k,b,r,o,w,n,f,x,j,m,p,s,v,l,a,z,y,d,g,{\textrm {space}}\}$. Of course the elements can be re-arranged in order $\{a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z,{\textrm {space}}\}$. The set also contains the space. For now let's consider spaces as an important component in our analysis. The sentence above is known to have all the letters in the alphabet. Remember, a random variable $X$ would also have a probability distribution associated with it. We can determine the probability based on the frequency of letters for the entire set. For example, ${\textrm {space}}$ appears $8$ times, the letter $o$ appears $4$ times, the letter $e$ appears $3$ times and so on. The total number of occurrences of each outcome is $43$. The probability distribution would be:

• $P(a)={\frac {1}{43}}=0.023$
• $P(b)={\frac {1}{43}}=0.023$
• $P(c)={\frac {1}{43}}=0.023$
• $P(d)={\frac {1}{43}}=0.023$
• $P(e)={\frac {3}{43}}=0.070$
• $P(f)={\frac {1}{43}}=0.023$
• $P(g)={\frac {1}{43}}=0.023$
• $P(h)={\frac {2}{43}}=0.047$
• $P(i)={\frac {1}{43}}=0.023$
• $P(j)={\frac {1}{43}}=0.023$
• $P(k)={\frac {1}{43}}=0.023$
• $P(l)={\frac {1}{43}}=0.023$
• $P(m)={\frac {1}{43}}=0.023$
• $P(n)={\frac {1}{43}}=0.023$
• $P(o)={\frac {4}{43}}=0.093$
• $P(p)={\frac {1}{43}}=0.023$
• $P(q)={\frac {1}{43}}=0.023$
• $P(r)={\frac {2}{43}}=0.047$
• $P(s)={\frac {1}{43}}=0.023$
• $P(t)={\frac {2}{43}}=0.047$
• $P(u)={\frac {2}{43}}=0.047$
• $P(v)={\frac {1}{43}}=0.023$
• $P(w)={\frac {1}{43}}=0.023$
• $P(x)={\frac {1}{43}}=0.023$
• $P(y)={\frac {1}{43}}=0.023$
• $P(z)={\frac {1}{43}}=0.023$
• $P({\textrm {space}})={\frac {8}{43}}=0.186$

We expanded the distribution for clarity.