Overview
The simplest example of a sparse model is typically considered to be OLS Regression with L1 Regularization.When a standard OLS regresion is run against a set of features, {% X_1, X_2,...,X_m %}, the regression will likely produce a non zero coefficient for each factor incldued in the regression.
L1 Regularization
The L1 Regularized OLS Regression is given by choosing a loss function
{% f(\vec{w}) = \sum_i^n (y_i - (w_0 + \textbf{w}^T \textbf{x}_i))^2 + \lambda || \textbf{w} ||_1 %}
where
{% || \textbf{w} ||_1 = \sum |w_i| %}
The L1 regularized regression is known to produce sparse solutions, where a number of the features to the regression
are assigned a zero coefficient. The number of features that are assigned a zero coefficient increases as
{% \lambda %} is increased.
Sample Code
import numpy as np
from sklearn.linear_model import Lasso
inputs = [[1.0,1.0],[1.2,2.0],[0.9,0.6]]
outputs = [[1],[2],[0.9]]
X = np.array(inputs)
y = np.array(outputs)
reg = Lasso(alpha=0.1)
reg.fit(X,y)
print(reg.intercept_)
print(reg.coef_)