Difference between revisions of "Entropy, Relative Entropy, Mutual Information"
Jump to navigation
Jump to search
| Line 8: | Line 8: | ||
=== Relative Entropy === | === Relative Entropy === | ||
* a measure of the distance between two distributions | * a measure of the distance between two distributions | ||
| + | * a measure of the inefficiency of assuming that the distribution is <math>q</math> when the true distribution is <math>p</math>. | ||
Revision as of 12:54, 25 June 2020
Definitions
Entropy
- a measure of the uncertainty of a random variable
- The entropy of a random variable is a measure of the uncertainty of the random variable
- it is a measure of the amount of information required on the average to describe the random variable
Relative Entropy
- a measure of the distance between two distributions
- a measure of the inefficiency of assuming that the distribution is when the true distribution is .
