Entropy, Relative Entropy, Mutual Information

Definitions

Entropy

• a measure of the uncertainty of a random variable
• The entropy of a random variable is a measure of the uncertainty of the random variable
  • it is a measure of the amount of information required on the average to describe the random variable

Relative Entropy

• a measure of the distance between two distributions
• a measure of the inefficiency of assuming that the distribution is $q$ when the true distribution is $p$.