Gradient Descent - Line Search
Overview
The line search method is a way to determine a step size in the
gradient descent
method. Once a gradient is computed for a function {% f %}, and a corresponding descent direction
{% d %}, then the line search seeks to find the optimal step size {% \eta %}.
It consructs a new function {% \phi(\eta) %} defined as
{% \phi(\eta) = f(\vec{\theta}_k + \eta \vec{d}_k) %}
and then uses any of a number of 1-d optimization techniques to find the optimal {% \eta %}.
One such method is to use the
Newton Raphson
method.
