Overview
The primary definition in information theory is Entropy. It provides a measure of how uncertain a random variable is. While there are other measures that could be deemed measures of uncertainty. (see variance for example)
Entropy is central to information theory because of its connection to data compression.
Entropy
The entropy of a random variable is defined as
{% H(X) = - \sum p(x) \times log [p(x)] %}
The entropy can be recast in terms of expectation.
{% H(X) = \mathbb{E} [log (1/p(x))] %}
The plot below shows the entropy of a bernoulli variable with probability p on the x axis. (the other probability is 1-p )
copy