Next Word Prediction Model

Most of the keyboards in smartphones give next word prediction features; google also uses next word prediction based on our browsing history. So a preloaded data is also stored in the keyboard function of our smartphones to predict the next word correctly. In this article, I will train a Deep Learning model for next word prediction using Python. I will use the Tensorflow and Keras library in Python for next word prediction model.

For making a Next Word Prediction model, I will train a Recurrent Neural Network (RNN). So let’s start with this task now without wasting any time.

Next Word Prediction Model

To start with our next word prediction model, let’s import some all the libraries we need for this task:

import numpy as np from nltk.tokenize import RegexpTokenizer from keras.models import Sequential, load_model from keras.layers import LSTM from keras.layers.core import Dense, Activation from keras.optimizers import RMSprop import matplotlib.pyplot as plt import pickle import heapq

As I told earlier, Google uses our browsing history to make next word predictions, smartphones, and all the keyboards that are trained to predict the next word are trained using some data. So I will also use a dataset. You can download the dataset from here.

Now let’s load the data and have a quick look at what we are going to work with:

path = '1661-0.txt' text = open(path).read().lower() print('corpus length:', len(text))

corpus length: 581887

Now I will split the dataset into each word in order but without the presence of some special characters.

tokenizer = RegexpTokenizer(r'w+') words = tokenizer.tokenize(text)
['project', 'gutenberg', 's', 'the', 'adventures', 'of', 'sherlock', 'holmes', 'by', ............................... , 'our', 'email', 'newsletter', 'to', 'hear', 'about', 'new', 'ebooks']

Now the next process will be performing the feature engineering in our data. For this purpose, we will require a dictionary with each word in the data within the list of unique words as the key, and it’s significant portions as value.

unique_words = np.unique(words) unique_word_index = dict((c, i) for i, c in enumerate(unique_words))

Feature Engineering

Feature Engineering means taking whatever information we have about our problem and turning it into numbers that we can use to build our feature matrix. If you want a detailed tutorial of feature engineering, you can learn it from here.

Here I will define a Word length which will represent the number of previous words that will determine our next word. I will define prev words to keep five previous words and their corresponding next words in the list of next words.

WORD_LENGTH = 5 prev_words = [] next_words = [] for i in range(len(words) - WORD_LENGTH): prev_words.append(words[i:i + WORD_LENGTH]) next_words.append(words[i + WORD_LENGTH]) print(prev_words[0]) print(next_words[0])
['project', 'gutenberg', 's', 'the', 'adventures']

Now I will create two numpy arrays x for storing the features and y for storing its corresponding label. I will iterate x and y if the word is available so that the corresponding position becomes 1.

X = np.zeros((len(prev_words), WORD_LENGTH, len(unique_words)), dtype=bool) Y = np.zeros((len(next_words), len(unique_words)), dtype=bool) for i, each_words in enumerate(prev_words): for j, each_word in enumerate(each_words): X[i, j, unique_word_index[each_word]] = 1 Y[i, unique_word_index[next_words[i]]] = 1

Now before moving forward, have a look at a single sequence of words:

print(X[0][0])

[False False False … False False False]

Building the Recurrent Neural network

As I stated earlier, I will use the Recurrent Neural networks for next word prediction model. Here I will use the LSTM model, which is a very powerful RNN.

model = Sequential() model.add(LSTM(128, input_shape=(WORD_LENGTH, len(unique_words)))) model.add(Dense(len(unique_words))) model.add(Activation('softmax'))

Training the Next Word Prediction Model

I will be training the next word prediction model with 20 epochs:

optimizer = RMSprop(lr=0.01) model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy']) history = model.fit(X, Y, validation_split=0.05, batch_size=128, epochs=2, shuffle=True).history

Now we have successfully trained our model, before moving forward to evaluating our model, it will be better to save this model for our future use.

model.save('keras_next_word_model.h5') pickle.dump(history, open("history.p", "wb")) model = load_model('keras_next_word_model.h5') history = pickle.load(open("history.p", "rb"))

Evaluating the Next Word Prediction Model

Now let’s have a quick look at how our model is going to behave based on its accuracy and loss changes while training:

plt.plot(history['acc']) plt.plot(history['val_acc']) plt.title('model accuracy') plt.ylabel('accuracy') plt.xlabel('epoch') plt.legend(['train', 'test'], loc='upper left')
next word prediction
plt.plot(history['loss']) plt.plot(history['val_loss']) plt.title('model loss') plt.ylabel('loss') plt.xlabel('epoch') plt.legend(['train', 'test'], loc='upper left')
image for post

Testing Next Word Prediction Model

Now let’s build a python program to predict the next word using our trained model. For this, I will define some essential functions that will be used in the process.

def prepare_input(text): x = np.zeros((1, SEQUENCE_LENGTH, len(chars))) for t, char in enumerate(text): x[0, t, char_indices[char]] = 1. return x

Now before moving forward, let’s test the function, make sure you use a lower() function while giving input :

prepare_input("This is an example of input for our LSTM".lower())
array([[[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
...,
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.]]])

Note that the sequences should be 40 characters (not words) long so that we could easily fit it in a tensor of the shape (1, 40, 57). Not before moving forward, let’s check if the created function is working correctly.

def prepare_input(text): x = np.zeros((1, WORD_LENGTH, len(unique_words))) for t, word in enumerate(text.split()): print(word) x[0, t, unique_word_index[word]] = 1 return x prepare_input("It is not a lack".lower())
array([[[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
...,
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.]]])

Now I will create a function to return samples:

def sample(preds, top_n=3): preds = np.asarray(preds).astype('float64') preds = np.log(preds) exp_preds = np.exp(preds) preds = exp_preds / np.sum(exp_preds) return heapq.nlargest(top_n, range(len(preds)), preds.take)

And now I will create a function for next word prediction:

def predict_completion(text): original_text = text generated = text completion = '' while True: x = prepare_input(text) preds = model.predict(x, verbose=0)[0] next_index = sample(preds, top_n=1)[0] next_char = indices_char[next_index] text = text[1:] + next_char completion += next_char if len(original_text + completion) + 2 > len(original_text) and next_char == ' ': return completion

This function is created to predict the next word until space is generated. It will do this by iterating the input, which will ask our RNN model and extract instances from it. Now I will modify the above function to predict multiple characters:

def predict_completions(text, n=3): x = prepare_input(text) preds = model.predict(x, verbose=0)[0] next_indices = sample(preds, n) return [indices_char[idx] + predict_completion(text[1:] + indices_char[idx]) for idx in next_indices]

Now I will use the sequence of 40 characters that we can use as a base for our predictions.

quotes = [ "It is not a lack of love, but a lack of friendship that makes unhappy marriages.", "That which does not kill us makes us stronger.", "I'm not upset that you lied to me, I'm upset that from now on I can't believe you.", "And those who were seen dancing were thought to be insane by those who could not hear the music.", "It is hard enough to remember my opinions, without also remembering my reasons for them!" ]

Now finally, we can use the model to predict the next word:

for q in quotes: seq = q[:40].lower() print(seq) print(predict_completions(seq, 5)) print()
it is not a lack of love, but a lack of
['the ', 'an ', 'such ', 'man ', 'present, ']
that which does not kill us makes us str
['ength ', 'uggle ', 'ong ', 'ange ', 'ive ']
i'm not upset that you lied to me, i'm u
['nder ', 'pon ', 'ses ', 't ', 'uder ']
and those who were seen dancing were tho
['se ', 're ', 'ugh ', ' servated ', 't ']it is hard enough to remember my opinion
[' of ', 's ', ', ', 'nof ', 'ed ']

Also Read: Data Augmentation in Deep Learning.

I hope you liked this article of Next Word Prediction Model, feel free to ask your valuable questions in the comments section below.

Follow Us:

Leave a Reply