Difference between revisions of "Entropy, Relative Entropy, Mutual Information"

From Microlab Classes
Jump to navigation Jump to search
Line 42: Line 42:
  
 
'''Proof''': Since <math>0 \le p\left(x\right) \le 1</math>, then <math>\log_2\left(\tfrac{1}{p\left(x\right)}\right) \ge 0</math>, and subsequently, <math>E\left[\log_2\left(\tfrac{1}{p\left(x\right)}\right)\right] \ge 0</math>. Thus from Eq. ({{EquationNote|4}}) we get <math>H\left(X\right)\ge 0</math>.
 
'''Proof''': Since <math>0 \le p\left(x\right) \le 1</math>, then <math>\log_2\left(\tfrac{1}{p\left(x\right)}\right) \ge 0</math>, and subsequently, <math>E\left[\log_2\left(\tfrac{1}{p\left(x\right)}\right)\right] \ge 0</math>. Thus from Eq. ({{EquationNote|4}}) we get <math>H\left(X\right)\ge 0</math>.
 +
 +
=== Lemma 2 ===
 +
<math>H_b\left(X\right)=\left(\log_b a\right)\cdot H_a\left(X\right)</math>

Revision as of 16:46, 25 June 2020

Definitions

Entropy

  • a measure of the uncertainty of a random variable
  • The entropy of a random variable is a measure of the uncertainty of the random variable
    • it is a measure of the amount of information required on the average to describe the random variable

Relative Entropy

  • a measure of the distance between two distributions
  • a measure of the inefficiency of assuming that the distribution is when the true distribution is .

Mutual Information

  • a measure of the amount of information that one random variable contains about another random variable

Entropy

Definitions:

  • a measure of the uncertainty of a random variable
  • The entropy of a random variable is a measure of the uncertainty of the random variable
    • it is a measure of the amount of information required on the average to describe the random variable

The entropy of a discrete random variable, , is

 

 

 

 

(1)

where has a probability mass function (pmf), , and an alphabet .

Expected Value

For a discrete random variable, , with probability mass function, , the expected value of is

 

 

 

 

(2)

For a discrete random variable, , with probability mass function, , the expected value of is

 

 

 

 

(3)

Consider the case where . We get

 

 

 

 

(4)

Lemma 1

Proof: Since , then , and subsequently, . Thus from Eq. (4) we get .

Lemma 2