Overview
A neural network layer is just a function that takes an input and produces an output. At a basic level, we will implement a layer as a simple Javascript object, that has a type (just a string that is used for human readability) and a function that takes an input and returns the output.
{
type:'name',
evaluate:function(input){},
}
The input provided should be a matrix or vector, following the conventions used in the linear algebra library. The output should also be a matrix or vector. (That is, all inputs and outputs should be composed of arrays of arrays. A scalar is represented as [[scalar]]).
Gradients
The backpropagation algorithm needs the gradient of each layer in order to compute the network parameter updates. There are two gradients to consider.
- The gradient of the layer with respect to its input - all layers will have a gradient with respect to its inputs, that is, how much the output changes when its inputs changes.
- The gradient of the layer with resept to its paramters. - only layers with parameters will have a gradient with respect to its parameters. For a simple neural network, only the linear transformations with weights have a gradient with respect to its weights. Activation functions do not (typically) have adjustable parameters, so they do not produce this gradient.
As a result, each layer should add on a function for each gradient that is relevant to that layer.
{
type:'name',
evaluate:function(input){},
inputGradient:function(input){},
//parameterGradient should only be present for a layer with tunable parameters
//activation functions should not include this function
parameterGradient:function(input){}
}
NOTE: in this example we follow the denominator layout. That is, the derivative of a scalar with respect to a column vector, should be a column vector.
In addition, we require that the gradient with respect to a matrix should be vectorized.