optimization

Optimization

Optimization is the process of finding the inputs at which a function hits its maximum or minimum values.

That is, given a function

{% y = f(x) %}

to find the maximum is the find the value of {% x' %} such that

{% f(x') \geq f(x) %}

for any other value of {% x %} in the domain.

Finding the minimum of a function is an equivalent problem to finding a maximum of the inverted function.

Analytic methods provide ways to find the extremes of a function with just methods from standard analysis (linear algebra and calculus)

Derivative Method: the standard method from calculus of finding the locations where the derivative of the function in question is equal to zero.
Lagrange Multipliers: is a multivariable calculus method for optimizing a function given a set of differentiable function constraints.

Random Search : Random search is a brute force method of randomly searching the input space for the optimal points. It is inefficient in finding points close to the optimal, but can be used to quickly reduce the input space to a space that can be subject to more fine tuned methods.
Golden Section : The golden section algorithm is an algorithm for finding an optimzal point of a single variable function that uses a bisection search method to quickly narrow the search space.
Linear and Non-linear Programming :
Gradient Descent : Gradient Descent is an algorithm for finding the minimum of a given mutlivariable function. It is a local algorithm, that is, it starts at a given point, and then examines the function value around that point (local area) in order to determine which point to evaluate next. Technically, it determines the rate of change of the function in each direction (along each axis of the inputs), determines which direction represents the fastest rate and then proceeds in that direction by a given step size.
Newtons Method : Newtons method uses the newton-raphson method for quickly finding a maximum. However, convergence is guaranteed under certain conditions, and often requires that the initial guess is "close" to the answer in some sense.
Genetic Algorithm : an algorithm inspired by natural selection and most often associated with machine learning.