Grid Search for Model Tuning

In this article, I will take you through a very powerful algorithm in Machine Learning, which is the Grid Search Algorithm. It is mostly used in hyperparameters tuning and models selection in Machine Learning. Here I will teach you how to implement this algorithm using python. At the end of this article, you will learn how to apply it in a real-life problem and how to choose the most effective hyperparameters for our Machine Learning model for great accuracy.

What is Grid Search?

Grid Search is used in Fine-tuning a Machine Learning model. Let’s assume that you have a shortlist of promising models. You now need to fine-tune them. So how you will do that?

One option would be to fiddle with the hyperparameters manually until you find a great combination of hyperparameter values. This would be very tedious work, and you may not have time to explore many combinations.

Instead, you can use the Grid Search Algorithm to look for you. All you need to do is tell it which hyperparameters you want it to experiment with and what values to try out, and it will use cross-validation to evaluate all the possible combinations of hyperparameters values.

You should also know that you can treat some of the data preparation steps as hyperparameters. For example, the grid search will automatically find out whether or not to add a feature you were not sure about. It may similarly be used to automatically find the best way to handle outliers, missing features, feature selection, and more.

Training a Deep Learning Model

Now let’s see how we can implement the Grid Search Algorithm in Deep Learning. The dataset I will use here is based on diabetes, which can be easily downloaded from here.

Now let’s start with importing the necessary libraries:

from sklearn.model_selection import GridSearchCV, KFold
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.wrappers.scikit_learn import KerasClassifier
from keras.optimizers import Adam
import sys
import pandas as pd
import numpy as npCode language: Python (python)

Now let’s import and read the dataset.

columns = ['num_pregnant', 'glucose_concentration', 'blood_pressure', 'skin_thickness',
           'serum_insulin', 'BMI', 'pedigree_function', 'age', 'class']

data_path = pd.read_csv("pima-indians-diabetes.csv")

df = pd.read_csv(data_path, names=columns)
df.head()Code language: Python (python)

Now let’s remove the unnecessary rows and replace the NaN values with 0.

# Remove first 9 non-data rows
df = df.iloc[9:]

# Replace NaN (Not a Number) values with 0 in each column
for col in columns:
    df[col].replace(0, np.NaN, inplace=True)

df.dropna(inplace=True) # Drop all rows with missing values
dataset = df.values # Convert dataframe to numpy arrayCode language: Python (python)

Now I will divide the dataset into features and labels by using the Standard Scalar method.

X = dataset[:,0:8]
Y = dataset[:, 8].astype(int)

# Normalize the data using sklearn StandardScaler
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler().fit(X)

# Transform and display the training data
X_standardized = scaler.transform(X)

data = pd.DataFrame(X_standardized)Code language: Python (python)

Now I will create function to create a deep learning model.

def create_model(learn_rate, dropout_rate):
    # Create model
    model = Sequential()
    model.add(Dense(8, input_dim=8, kernel_initializer='normal', activation='relu'))
    model.add(Dropout(dropout_rate))
    model.add(Dense(4, input_dim=8, kernel_initializer='normal', activation='relu'))
    model.add(Dropout(dropout_rate))
    model.add(Dense(1, activation='sigmoid'))

    # Compile the model
    adam = Adam(lr=learn_rate)
    model.compile(loss='binary_crossentropy', optimizer=adam, metrics=['accuracy'])
    return modelCode language: Python (python)

Now, I will implement a grid search algorithm but to understand it better let’s first train our model without implementing it.

# Declare parameter values
dropout_rate = 0.1
epochs = 1
batch_size = 20
learn_rate = 0.001

# Create the model object by calling the create_model function we created above
model = create_model(learn_rate, dropout_rate)

# Fit the model onto the training data
model.fit(X_standardized, Y, batch_size=batch_size, epochs=epochs, verbose=1)Code language: Python (python)

Epoch 1/1
130/130 [==============================] - 0s 2ms/step - loss: 0.6934 - accuracy: 0.6000

So, we have got an accuracy of 60 percent without implementing the grid search algorithm let’s see how much we can improve the accuracy after implementing the algorithm in our deep learning model.

Implementing Grid Search Algorithm

First I will modify the function that I created above because to use our Algorithm we need to apply some parameters to the create_model function.

def create_model(learn_rate, dropout_rate):
    # Create model
    model = Sequential()
    model.add(Dense(8, input_dim=8, kernel_initializer='normal', activation='relu'))
    model.add(Dropout(dropout_rate))
    model.add(Dense(4, input_dim=8, kernel_initializer='normal', activation='relu'))
    model.add(Dropout(dropout_rate))
    model.add(Dense(1, activation='sigmoid'))

    # Compile the model
    adam = Adam(lr=learn_rate)
    model.compile(loss='binary_crossentropy', optimizer=adam, metrics=['accuracy'])
    return model

# Create the model
model = KerasClassifier(build_fn=create_model, verbose=1)Code language: Python (python)

Now let’s apply our Algorithm and fit the data on it:

# Define the parameters that you wish to use in your Grid Search along
# with the list of values that you wish to try out
learn_rate = [0.001, 0.02, 0.2]
dropout_rate = [0.0, 0.2, 0.4]
batch_size = [10, 20, 30]
epochs = [1, 5, 10]

seed = 42

# Make a dictionary of the grid search parameters
param_grid = dict(learn_rate=learn_rate, dropout_rate=dropout_rate, batch_size=batch_size, epochs=epochs )

# Build and fit the GridSearchCV
grid = GridSearchCV(estimator=model, param_grid=param_grid,
                    cv=KFold(random_state=seed), verbose=10)

grid_results = grid.fit(X_standardized, Y)

# Summarize the results in a readable format
print("Best: {0}, using {1}".format(grid_results.best_score_, grid_results.best_params_))

means = grid_results.cv_results_['mean_test_score']
stds = grid_results.cv_results_['std_test_score']
params = grid_results.cv_results_['params']

for mean, stdev, param in zip(means, stds, params):
    print('{0} ({1}) with: {2}'.format(mean, stdev, param))Code language: Python (python)

Best: 0.7959183612648322, using {'batch_size': 10, 'dropout_rate': 0.2, 'epochs': 10, 'learn_rate': 0.02}

In the output, you can see it provides us with the best hyperparameters combination which improves the accuracy of our model that is 75 percent.

Also, Read – GPUs Can Speed Up your Models.

I hope you liked this article on the Grid Search Algorithm in Deep Learning. Feel free to ask your valuable questions in the comments section below.

Follow Us: