Kullback Liebler

Overview


The Kullback Liebler function measures a "distance" between two statistical distributions. It is always {% \geq 0 %} and is zero only if the two distributions are the same. However, it is not symmetric and does not satisfy the triangle inequality. metric

Kullback Leibler Information


The Kullback Leibler distance is defined as an expectation with respect to one distribution of a specified random variable. In the discrete case, it is defined as
{% d = \mathbb{E}_g[log(\frac{g(x)}{f(x)})] %}
which for the continuous case becomes
{% d = \int log(\frac{g(x)}{f(x)}) g(x) dx %}

Properties


  • {% d(f, g) \geq 0 %}
  • {% d(f, g) = 0 \rightarrow g=f %}

Contents