Information Theory

Overview

The theory of information takes several different forms. This corner focuses on the theory of information initially developed by Claude Shannon.

Topics

• Entropy - the central notion in information theory, it is a measure of uncertainty or information
• Joint Entropy - the amount of information one random has about another
• Conditional Entropy - is a measure of the additional amount of uncertainty that one variable contributes to the join entropy.
• Entropy Rate - the entropy of a stochastic process

Information Theory, Statistics and Artificial Intelligence

• Entropy Optimization is a principle which has broad applicability in statistics and machine learning.