Entropy
Overview
The primary definition in information theory is Entropy. It provides a measure of how uncertain a random variable is.
While there are other measures that could be deemed measures of uncertainty
(see
variance for example),
, entropy is central to information theory
because of its connection to
data compression.
Entropy
The entropy of a random variable is defined as
{% H(X) = - \sum p(x) \times log [p(x)] %}
The entropy can be recast in terms of expectation.
{% H(X) = \mathbb{E} [log (1/p(x))] %}
The plot below shows the entropy of a
bernoulli variable
with probability p on the x axis. (the other probability is 1-p )
copy
