Difference between revisions of "Entropy, Relative Entropy, Mutual Information"

Revision as of 13:36, 25 June 2020

a measure of the uncertainty of a random variable
The entropy of a random variable is a measure of the uncertainty of the random variable
- it is a measure of the amount of information required on the average to describe the random variable

a measure of the distance between two distributions
a measure of the inefficiency of assuming that the distribution is $q$ when the true distribution is $p$ .

a measure of the amount of information that one random variable contains about another random variable

The entropy of a discrete random variable, $X$ , is

H\left(X\right)=-\sum _{x\in {\mathcal {X}}}p\left(x\right)\log _{2}\left(p\left(x\right)\right)

(1)

where $X$ has a probability mass function (pmf), $p\left(x\right)$ , and an alphabet ${\mathcal {X}}$ .

Given a random variable, $X$ with probability mass function $p\left(x\right)$ , the expected value of $g\left(X\right)$ is

@@ Line 20: / Line 20: @@
 where <math>X</math> has a probability mass function (pmf), <math>p\left(x\right)</math>, and an alphabet <math>\mathcal{X}</math>.
-=== Expectation of <math>g\left(X\right)</math> ===
+=== Expected Value ===
+Given a random variable, <math>X</math> with probability mass function <math>p\left(x\right)</math>, the expected value of <math>g\left(X\right)</math> is