Overview
Regressions are learning algoithms where the label assigned to any data point is taken from a continuous range of numeric values, such as from the real numbers. A simple example would be a forecast of an asset return.Regression Function
Given a distribution which generates data points of the form {% (x, y) %} the regression function can be defined to be
{% f_\rho(x) = \int_Y y \, d \rho(y|x) %}
The function takes a value of {% x %} as an input (note, x need not be a scalar)
returns the best estimate of y, where best estimate in this sense is the
conditional expectation
(see cucker)
This form of the regression function can be fit using a square error loss function. When a different loss function is used, the function being fit will not exactly correspond to the conditional expectation based function defined above.
Types
Given a dataset, and an assumption about the distribution generating the data, it is common practice to infer the parameters of the distribution using Maximum Likelihood methods.The following detail the methods used in several common cases.