bias variance derivation

Bias Variance Tradeoff Derivation

{% \mathbb{E}_{out}(g^D) = \mathbb{E}_x[(g^D(x) - f(x))^2] %}

{% \mathbb{E}_D [\mathbb{E}_{out}(g^D)] = \mathbb{E}_D[\mathbb{E}_x[(g^D(x) - f(x))^2]] %}

{% = \mathbb{E}_x[\mathbb{E}_D[(g^D(x) - f(x))^2]] %}

by exchanging the order of integration (Fubini)

{% = \mathbb{E}_x[\mathbb{E}_D(g^D(x)^2) -2 \mathbb{E}_D[g^D(x)]f(x) + f(x)^2] %}

Denote {% \bar{g} = \mathbb{E}_D[g^D(x)] %}

{% \mathbb{E}_D[\mathbb{E}_{out}(g^D)] %}

{% = \mathbb{E}_x[\mathbb{E}_D[g^D(x)^2] - 2\bar{g}(x)f(x) + f(x)^2] %}

{% = \mathbb{E}_x[\mathbb{E}_D[g^D(x)^2] - \bar{g}(x)^2 + \bar{g}(x)^2 - 2\bar{g}(x)f(x) + f(x)^2] %}

{% \mathbb{E}_D [\mathbb{E}_{out}(g^D)] = \mathbb{E}_x[bias(x)+var(x)] %}