ABSTRACT
Principal components analysis is a popular method for finding low-dimensional linear structure in higher dimensional data. It has a range of purposes, but let’s see how it can be helpful in regression problems. Designed experiments typically have predictors which are mutually orthogonal. This makes the fitting and interpretation of these models simpler. For observational data, predictors are often substantially correlated. Ridge regression makes the assumption that the regression coefficients (after normalization) should not be very large. Hence shrinkage is embedded in the method. Ridge regression is particularly effective when the model matrix is collinear and the usual least squares estimates of β appear to be unstable. The Orthogonal Matching Pursuit method goes further than lasso in just encouraging the elimination of predictors — it specifies a maximum number of nonzero coefficients. The scikit-learn package also contains a Bayesian regression implementation.
