In Machine Learning, Ridge and Lasso Regression are regularization techniques used in linear regression to prevent overfitting and improve the model’s generalization to unseen data. They work by adding a penalty term to the linear regression loss function. If you want to learn about Ridge and Lasso regression algorithms, this article is for you. In this article, I’ll take you through Ridge and Lasso regression and how to implement them using Python.
Ridge and Lasso Regression
Ridge and Lasso Regression are techniques that add regularization to linear regression models. Ridge encourages small coefficients and can prevent multicollinearity, while Lasso can perform feature selection by setting some coefficients to zero. Let’s explore both these algorithms in detail one by one.
Ridge Regression
Ridge Regression, also known as L2 regularization, adds a penalty term proportional to the square of the magnitude of the coefficients. It encourages the model to have smaller but non-zero coefficients. It helps in reducing the impact of high-value coefficients and, in turn, the risk of overfitting.
Ridge regression handles the problem of multicollinearity, which occurs when predictor variables are highly correlated. Multicollinearity refers to the situation where predictor variables are highly correlated with each other. It can cause issues in traditional linear regression, such as unstable and unreliable coefficient estimates. Ridge Regression helps mitigate this problem by reducing the impact of multicollinearity.
The mathematical formula for ridge regression is 尾 = (X^T X + 位I)^-1 X^T y, where 尾 is the vector of coefficients, X is the matrix of independent variables, y is the vector of values of dependent variables, 位 is the penalty parameter, and I is the identity matrix.
Ridge Regression works by introducing a penalty term, controlled by the regularization parameter 位, to the ordinary least squares equation. This penalty term restricts the magnitude of the coefficient estimates, thereby reducing their sensitivity to small changes in the input data. By adding this penalty, Ridge Regression shrinks the coefficients towards zero, but they do not become exactly zero. It allows the algorithm to handle multicollinearity and provide more stable and reliable predictions.
Lasso Regression
Lasso Regression, also known as L1 regularization, adds a penalty term proportional to the absolute values of the coefficients. It is a type of regression algorithm used for feature selection and regularization. It is similar to ridge regression, but it adds an L1 penalty term to the regression equation, resulting in a sparse model where some of the coefficients are set to zero.
The mathematical formula for lasso regression is 尾 = argmin(危(yi – 尾0 – 危j=1 to p xi,j尾j)^2 + 位危|尾j|), where 尾 is the vector of coefficients, 尾0 is the intercept term, xi,j is the value of the jth independent variable for the ith observation, yi is the value of the dependent variable for the ith observation, 位 is the parameter of penalty, and p is the number of independent variables.
In Lasso regression, the goal is to minimize the sum of squared residuals, just like in ordinary linear regression. However, Lasso introduces an additional term called the L1 penalty or the absolute value of the coefficients.
The L1 penalty encourages the coefficients of less important features to become exactly zero, effectively performing automatic feature selection. This characteristic sets Lasso regression apart from Ridge regression, where the coefficients are only shrunk towards zero but not exactly zero.
To achieve this, Lasso regression adjusts the coefficients during the model fitting process. As the penalty term increases, some coefficients are driven to zero, effectively excluding the corresponding features from the model.
Implementation of Ridge and Lasso Regression
Now, let’s see how to implement Ridge and Lasso Regression using Python. To implement these algorithms, I will use the diabetes data from scikit-learn. Here’s how to implement Ridge and Lasso Regression using Python:
from sklearn import datasets
from sklearn.linear_model import Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
# Load the diabetes dataset
diabetes = datasets.load_diabetes()
X = diabetes.data
y = diabetes.target
# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=0.2,
random_state=42)
# Ridge Regression Implementation
ridge = Ridge(alpha=1.0)
ridge.fit(X_train, y_train)
# Lasso Regression Implementation
lasso = Lasso(alpha=1.0)
lasso.fit(X_train, y_train)
# Make predictions
y_pred_ridge = ridge.predict(X_test)
y_pred_lasso = lasso.predict(X_test)
# Evaluate the models
mse_ridge = mean_squared_error(y_test, y_pred_ridge)
mse_lasso = mean_squared_error(y_test, y_pred_lasso)
print("Ridge Regression MSE:", mse_ridge)
print("Lasso Regression MSE:", mse_lasso)Ridge Regression MSE: 3077.41593882723 Lasso Regression MSE: 3403.5757216070733
In this example, we imported the diabetes dataset, split it into training and testing sets, implemented Ridge and Lasso Regression models, made predictions, and evaluated the models using mean squared error.
We used the alpha parameter in both algorithms. Alpha (位) is the regularization strength parameter. It controls the amount of regularization applied to the model. Larger values of alpha result in stronger regularization.
Here’s how we can visualize the results of both the Ridge and Lasso regression models that we trained above:
import plotly.express as px
import pandas as pd
# Create a DataFrame to store the results
results = pd.DataFrame({'Actual': y_test, 'Predicted_Ridge': y_pred_ridge,
'Predicted_Lasso': y_pred_lasso})
# visualize the actual vs. predicted values for Ridge Regression
fig_ridge = px.scatter(results, x='Actual', y='Predicted_Ridge',
title='Ridge Regression: Actual vs. Predicted',
labels={'Actual': 'Actual Values',
'Predicted_Ridge': 'Predicted Values'},
trendline='ols')
# visualize the actual vs. predicted values for Lasso Regression
fig_lasso = px.scatter(results, x='Actual', y='Predicted_Lasso',
title='Lasso Regression: Actual vs. Predicted',
labels={'Actual': 'Actual Values',
'Predicted_Lasso': 'Predicted Values'},
trendline='ols')
fig_ridge.show()
fig_lasso.show()
So, this is how Ridge and Lasso algorithms work.
Summary
So, Ridge and Lasso Regression are techniques that add regularization to linear regression models. Ridge encourages small coefficients and can prevent multicollinearity, while Lasso can perform feature selection by setting some coefficients to zero. I hope you liked this article on Ridge and Lasso regression algorithms in Machine Learning. Feel free to ask valuable questions in the comments section below.
