Grid Search for Model Tuning

In this article, I will take you through a very powerful algorithm in Machine Learning, which is the Grid Search Algorithm. It is mostly used in hyperparameters tuning and models selection in Machine Learning. Here I will teach you how to implement this algorithm using python. At the end of this article, you will learn how to apply it in a real-life problem and how to choose the most effective hyperparameters for our Machine Learning model for great accuracy.

What is Grid Search?

Grid Search is used in Fine-tuning a Machine Learning model. Let’s assume that you have a shortlist of promising models. You now need to fine-tune them. So how you will do that?

One option would be to fiddle with the hyperparameters manually until you find a great combination of hyperparameter values. This would be very tedious work, and you may not have time to explore many combinations.

Instead, you can use the Grid Search Algorithm to look for you. All you need to do is tell it which hyperparameters you want it to experiment with and what values to try out, and it will use cross-validation to evaluate all the possible combinations of hyperparameters values.

You should also know that you can treat some of the data preparation steps as hyperparameters. For example, the grid search will automatically find out whether or not to add a feature you were not sure about. It may similarly be used to automatically find the best way to handle outliers, missing features, feature selection, and more.

Training a Deep Learning Model

Now let’s see how we can implement the Grid Search Algorithm in Deep Learning. The dataset I will use here is based on diabetes, which can be easily downloaded from here.

Now let’s start with importing the necessary libraries:

from sklearn.model_selection import GridSearchCV, KFold from keras.models import Sequential from keras.layers import Dense, Dropout from keras.wrappers.scikit_learn import KerasClassifier from keras.optimizers import Adam import sys import pandas as pd import numpy as np
Code language: Python (python)

Now let’s import and read the dataset.

columns = ['num_pregnant', 'glucose_concentration', 'blood_pressure', 'skin_thickness', 'serum_insulin', 'BMI', 'pedigree_function', 'age', 'class'] data_path = pd.read_csv("pima-indians-diabetes.csv") df = pd.read_csv(data_path, names=columns) df.head()
Code language: Python (python)
grid search

Now let’s remove the unnecessary rows and replace the NaN values with 0.

# Remove first 9 non-data rows df = df.iloc[9:] # Replace NaN (Not a Number) values with 0 in each column for col in columns: df[col].replace(0, np.NaN, inplace=True) df.dropna(inplace=True) # Drop all rows with missing values dataset = df.values # Convert dataframe to numpy array
Code language: Python (python)

Now I will divide the dataset into features and labels by using the Standard Scalar method.

X = dataset[:,0:8] Y = dataset[:, 8].astype(int) # Normalize the data using sklearn StandardScaler from sklearn.preprocessing import StandardScaler scaler = StandardScaler().fit(X) # Transform and display the training data X_standardized = scaler.transform(X) data = pd.DataFrame(X_standardized)
Code language: Python (python)

Now I will create function to create a deep learning model.

def create_model(learn_rate, dropout_rate): # Create model model = Sequential() model.add(Dense(8, input_dim=8, kernel_initializer='normal', activation='relu')) model.add(Dropout(dropout_rate)) model.add(Dense(4, input_dim=8, kernel_initializer='normal', activation='relu')) model.add(Dropout(dropout_rate)) model.add(Dense(1, activation='sigmoid')) # Compile the model adam = Adam(lr=learn_rate) model.compile(loss='binary_crossentropy', optimizer=adam, metrics=['accuracy']) return model
Code language: Python (python)

Now, I will implement a grid search algorithm but to understand it better let’s first train our model without implementing it.

# Declare parameter values dropout_rate = 0.1 epochs = 1 batch_size = 20 learn_rate = 0.001 # Create the model object by calling the create_model function we created above model = create_model(learn_rate, dropout_rate) # Fit the model onto the training data model.fit(X_standardized, Y, batch_size=batch_size, epochs=epochs, verbose=1)
Code language: Python (python)
Epoch 1/1
130/130 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.6000

So, we have got an accuracy of 60 percent without implementing the grid search algorithm let’s see how much we can improve the accuracy after implementing the algorithm in our deep learning model.

Implementing Grid Search Algorithm

First I will modify the function that I created above because to use our Algorithm we need to apply some parameters to the create_model function.

def create_model(learn_rate, dropout_rate): # Create model model = Sequential() model.add(Dense(8, input_dim=8, kernel_initializer='normal', activation='relu')) model.add(Dropout(dropout_rate)) model.add(Dense(4, input_dim=8, kernel_initializer='normal', activation='relu')) model.add(Dropout(dropout_rate)) model.add(Dense(1, activation='sigmoid')) # Compile the model adam = Adam(lr=learn_rate) model.compile(loss='binary_crossentropy', optimizer=adam, metrics=['accuracy']) return model # Create the model model = KerasClassifier(build_fn=create_model, verbose=1)
Code language: Python (python)

Now let’s apply our Algorithm and fit the data on it:

# Define the parameters that you wish to use in your Grid Search along # with the list of values that you wish to try out learn_rate = [0.001, 0.02, 0.2] dropout_rate = [0.0, 0.2, 0.4] batch_size = [10, 20, 30] epochs = [1, 5, 10] seed = 42 # Make a dictionary of the grid search parameters param_grid = dict(learn_rate=learn_rate, dropout_rate=dropout_rate, batch_size=batch_size, epochs=epochs ) # Build and fit the GridSearchCV grid = GridSearchCV(estimator=model, param_grid=param_grid, cv=KFold(random_state=seed), verbose=10) grid_results = grid.fit(X_standardized, Y) # Summarize the results in a readable format print("Best: {0}, using {1}".format(grid_results.best_score_, grid_results.best_params_)) means = grid_results.cv_results_['mean_test_score'] stds = grid_results.cv_results_['std_test_score'] params = grid_results.cv_results_['params'] for mean, stdev, param in zip(means, stds, params): print('{0} ({1}) with: {2}'.format(mean, stdev, param))
Code language: Python (python)
Best: 0.7959183612648322, using {'batch_size': 10, 'dropout_rate': 0.2, 'epochs': 10, 'learn_rate': 0.02}

In the output, you can see it provides us with the best hyperparameters combination which improves the accuracy of our model that is 75 percent.

Also, Read – GPUs Can Speed Up your Models.

I hope you liked this article on the Grid Search Algorithm in Deep Learning. Feel free to ask your valuable questions in the comments section below.

Follow Us:

Default image
Aman Kharwal

I am a programmer from India, and I am here to guide you with Data Science, Machine Learning, Python, and C++ for free. I hope you will learn a lot in your journey towards Coding, Machine Learning and Artificial Intelligence with me.

Leave a Reply