Neural Networks Hyperparameters

Overview

When constructing a neural network, there are several choices that need to be made. In particular, you will have to select

• Number of Layers
• Number of Neurons per Layers
• Learning Rate (when applicable)
• Number of Iterations, Epochs

These parameters are called the hyper-parameters. (this is in contrast to the neurons weights which are referred to as parameters) There is no exact way to know what the best values of these hyper-parameters ahead of time. There are some rules of thumb, but ultimately one may need to optimize over the set of hyper-parameters as well as optimizing the model for each set.