Ridge Regression
The ridge regression method is similar to the least squares procedure
except that it penalizes the sizes of the regression coefficients.
Ridge regression is one of the most commonly used methods to overcome
data multicollinearity.
Details
Let 
be a vector of input variables and
be the response. For each 
,
the ridge regression model has the form similar to the linear
regression model [Hoerl70], except that the coefficients are estimated by minimizing a
different objective function [James2013]:
be a vector of input variables and
be the response. For each ,
the ridge regression model has the form similar to the linear
regression model [Hoerl70], except that the coefficients are estimated by minimizing a
different objective function [James2013]:
Here 
, 
, are referred to as independent
variables, and 
are referred to as dependent variables
or responses.
, , are referred to as independent
variables, and are referred to as dependent variables
or responses.Training Stage
Let 
be a set of training data, 
. The matrix 
contains observations 
, 
, 
, of
independent variables.
be a set of training data, . The matrix X
of size
contains observations , , , of
independent variables.For each 
, 
, the ridge regression estimates
by minimizing the
objective function:
, , the ridge regression estimates
by minimizing the
objective function:
are ridge parameters Prediction Stage
Ridge regression based prediction is done for input vector using the
equation 
for each .
using the
equation for each .