Javascript Neural Networks - Activation Functions

Overview


Adding an Activation Function


Any layer in the neural network model must be implemented as a Javascript object with two functions on it.

  • evaluate(input) - takes a set of inputs and returns the value of the layer applied to the inputs. The inputs are structured as a column vector.
  • inputGradient(input) - computes the gradient of this layer with respect to its inputs. This will be a Jacobian Matrix specified in denominator format. 



function relu(){
  let layer = {
    type:'relu',
    evaluate:function(input){
      let result = [];
      for(let row of input){
        let val = row[0];
        if(val >0) result.push([val]);
        else result.push([0]);
      }
      return result;
    },
    inputGradient:function(input){
      let result = [];
      for(let i=0;i<input.length;i++){
        let row = input[i];
        let val = row[0];
        let row2 = input.map(p=>0);
        if(val > 0) row2[i] = 1;
        result.push(row2);
      }      
      return result;
    }
  };
  return layer;
}