Difference between revisions of "Entropy, Relative Entropy, Mutual Information"
Jump to navigation
Jump to search
| Line 10: | Line 10: | ||
* a measure of the inefficiency of assuming that the distribution is <math>q</math> when the true distribution is <math>p</math>. | * a measure of the inefficiency of assuming that the distribution is <math>q</math> when the true distribution is <math>p</math>. | ||
| − | == Mutual Information == | + | === Mutual Information === |
| − | * a measure of the amount of information that one random variable contains about another random variable | + | * a measure of the amount of information that one random variable contains about another random variable |
Revision as of 12:56, 25 June 2020
Definitions
Entropy
- a measure of the uncertainty of a random variable
- The entropy of a random variable is a measure of the uncertainty of the random variable
- it is a measure of the amount of information required on the average to describe the random variable
Relative Entropy
- a measure of the distance between two distributions
- a measure of the inefficiency of assuming that the distribution is when the true distribution is .
Mutual Information
- a measure of the amount of information that one random variable contains about another random variable
