Overview
The bias variance tradeoff discusses how the accuracy and error of a model is related to the model complexity.Variations of the bias variance tradeoff can be found within mathematical statistics, see bias variance in stastistical inference for example.
The discussion here follows the logic in Sergios Theodoridis.
Definitions and Derivation
We start by defining the mean square error of the estimator as:
{% MSE = \mathbb{E}[(\hat{\theta} - \theta)^2] %}
which can be expanded to
{% MSE = \mathbb{E}[\{(\hat{\theta} - \mathbb{E}[\hat{\theta}])
+ (\mathbb{E}[\hat{\theta} - \theta])
\}^2] %}
This can be described as the sum of two terms, the variance (first term) and the bias (second term).
{%
MSE = \mathbb{E}[(\hat{\theta} - \mathbb{E}[\hat{\theta}])^2] +
(\mathbb{E}[\hat{\theta}] - \theta)^2
%}
Implications
The primary implication of the bias variance tradeoff is that if one insists on an estimator where the bias is zero, one may end up with an estimator that does not minimize the mean squared error. That is
{% min \hat{\theta} %}
where you insist on the following constraint
{% \mathbb{E}[\hat{\theta}] = \theta %}
may not achieve the same MSE as minimizing without the constraint.