Maximum Likelihood

Overview


This example demonstrates calculating a maximum likelihood estimate of a normal distribution given a set of samples from the distribution.

We assume we have a number of data points. Each point has 4 properties, 3 factors and a value. 


let sample = {
	factor1:0.2,
	factor2:1.2,
	factor3:1,0,
	value:3.4
}


We assume that the value is drawn from a normal distribution
{% f(x) = \sqrt{1/2\pi \sigma^2 } \times e ^{-0.5 [(x-\mu)/\sigma]^2} %}
Here we assume that {% \mu %} and {% \sigma %} are not constant, rather they are functions of the 3 factors.

Maximum Likelihood


The likelihood of the drawn sample of points is.
{% Likelihood = \Pi \sqrt{1/2\pi \sigma^2 } \times e ^{-0.5 [(x-\mu)/\sigma]^2} %}
The log likelihood is then given by
{% logLikelihood = \sum log[ \sqrt{1/2\pi \sigma^2 } \times e ^{-0.5 [(x-\mu)/\sigma]^2} ] %}
{% = \sum log(\sqrt{1/2\pi \sigma^2}) + log[ e ^{-0.5 [(x-\mu)/\sigma]^2} ] %}
{% = \sum log(\sqrt{1/2\pi \sigma^2}) + -0.5 [(x-\mu)/\sigma]^2 ] %}
A neural network can be constructed to forecast the {% \mu %} and {% \sigma %} of the distribution based on the 3 factors. In order to do this, the neural network must have 3 inputs (one for each factor) and two outputs, representing {% \mu %} and {% \sigma %}.

Lastly, a loss function must be chosen. Here there is a bit of complexity. Most api's assume that you provide a target value that you can compare the predicted value to. In this case, the target values are unknown. Rather, we have to provide a loss function which is the negative log likelihood.

Implementation


The following implementation uses the numeric neural network library. 


function loss(){
    let weights = [...arguments]
    let sum = 0;
    for(let item of sample){
        let predicted = network(item.factors, ...weights);
        let mean = predicted[0][0];
        let stdDev = predicted[1][0];
        sum += Math.log(Math.sqrt(1/(2*Math.PI*stdDev*stdDev)));
        let temp = (item.value - mean)/stdDev;
        sum += -0.5 * temp*temp;
    }
    return -1*sum;
}


For the example, we generate a random sample of points. We arbitrarily choose a function of the factors to represent the mean and standard deviation. 


let sample = [];

for(let i=0;i<100;i++){
    let point = [Math.random(),Math.random(),Math.random()];
    let item = {
        factors:point.map(p=>[p]),
        mean:point[0]*point[0]+ point[1] - 0.5*point[2],
        stdDev: 0.5*point[1] + 0.2 * point[2]
    };
    
    item.value = nm.random(item.mean, item.stdDev);
    sample.push(item);
}
Try it!

Contents