Maximum Likelihood
Overview
Maximum Likelihood is a method for estimating the parameters of some known distribution given a set of data points generated
by that distribution. This is accomplished by maximizing the Likelihood function for the distribution and given dataset.
Likelihood Function Definition
Given a
distribution density function
{% f %} and a set of data points, {% x_1, ... ,x_n %}
The Likelihood function is defined as
{% Likelihood(\theta) = f(x_1,...,x_n | \theta) %}
The maximum Likelihood method maximizes the likelihood function over the range of possible {% \theta %}.
Identical Independent Variables
When the variables are independent and identically distributed, the equation for {% f %} can be simplified by
defining the function
{% g %} to be the single variable density function that each variable is drawn from.
{% Likelihood(\theta) = \Pi_{i=1}^n g(X_i|\theta) %}
In this case, the function can be simplified by taking the logarithm
{% Log \, Likelihood = \sum_{i=1}^n log[g(X_i|\theta)] %}
Because the logarithm is a monotonically increasing function, the maximum of the Likelihood function is achieved at the
same point of the maximum of the log Likelihood.
The
standard method
of finding the optimal point is to then take the
derivative
of the log-likelihood function and
setting it to zero.
(see
derivative method of optimization)
{% \sum_{i=1}^n \frac{dlog[g(X_i|\theta)]}{dx} = 0 %}
