Overview
Adding an Activation Function
Any layer in the neural network model must be implemented as a Javascript object with two functions on it.- evaluate(input) - takes a set of inputs and returns the value of the layer applied to the inputs. The inputs are structured as a column vector.
- inputGradient(input) - computes the gradient of this layer with respect to its inputs. This will be a Jacobian Matrix specified in denominator format.
function relu(){
let layer = {
type:'relu',
evaluate:function(input){
let result = [];
for(let row of input){
let val = row[0];
if(val >0) result.push([val]);
else result.push([0]);
}
return result;
},
inputGradient:function(input){
let result = [];
for(let i=0;i<input.length;i++){
let row = input[i];
let val = row[0];
let row2 = input.map(p=>0);
if(val > 0) row2[i] = 1;
result.push(row2);
}
return result;
}
};
return layer;
}