More Deep Learning Projects…

Project 20: A Next Word Prediction Model with Deep Learning

Most smartphone keyboards offer next word prediction features; Google also uses next word prediction based on our browsing history. So preloaded data is also stored in our smartphones' keyboard function to predict the next word correctly. In this article, I will train a deep learning model to predict the next word with Python. I will use the TensorFlow and Keras libraries in Python for the next word prediction model.

To make a next word prediction model, I will train a recurrent neural network (RNN). Let's get started with this task now without wasting any time.

The next word prediction model

To get started with the next word prediction model, let's import some libraries we need for this task:

import numpy as np
from nltk.tokenize import RegexpTokenizer
from keras.models import Sequential, load_model
from keras.layers import LSTM
from keras.layers.core import Dense, Activation
from keras.optimizers import RMSprop
import matplotlib.pyplot as plt
import pickle
import heapq

As I said earlier, Google uses our browsing history to make next word predictions, and smartphones and all keyboards trained to predict the next word are trained on some data. So I will also use a dataset. You can download the dataset from here.

Now let's load the data and take a quick look at what we will be working with:

path = '1661-0.txt'
text = open(path).read().lower()
print('corpus length:', len(text))

Now I will split the dataset into individual words in order, but without some special characters.

tokenizer = RegexpTokenizer(r'w+')
words = tokenizer.tokenize(text)

['project', 'gutenberg', 's', 'the', 'adventures', 'of', 'sherlock', 'holmes', 'by', ............................... , 'our', 'email', 'newsletter', 'to', 'hear', 'about', 'new', 'ebooks']

Now the next step is to perform feature engineering on our data. For this purpose, we will need a dictionary with each word in the data within the list of unique words as the key, and its important parts as the value.

unique_words = np.unique(words)
unique_word_index = dict((c, i) for i, c in enumerate(unique_words))

Feature engineering

Feature engineering means taking whatever information we have about our problem and turning it into numbers we can use to build our feature matrix. If you want a detailed tutorial on feature engineering, you can learn it from here.

Here I will define a word length that represents the number of previous words that will determine our next word. I will define the previous words to keep the five previous words and their corresponding next words in the list of next words.

WORD_LENGTH = 5
prev_words = []
next_words = []
for i in range(len(words) - WORD_LENGTH):
    prev_words.append(words[i:i + WORD_LENGTH])
    next_words.append(words[i + WORD_LENGTH])
print(prev_words[0])
print(next_words[0])

Now I will create two empty arrays: x to store the features and y to store their corresponding labels. I will iterate over x and y so that, if the word is available, the corresponding position becomes 1.

X = np.zeros((len(prev_words), WORD_LENGTH, len(unique_words)), dtype=bool)
Y = np.zeros((len(next_words), len(unique_words)), dtype=bool)
for i, each_words in enumerate(prev_words):
    for j, each_word in enumerate(each_words):
        X[i, j, unique_word_index[each_word]] = 1
    Y[i, unique_word_index[next_words[i]]] = 1

Building the recurrent neural network

 

As I mentioned earlier, I will use recurrent neural networks for the next word prediction model. Here I will use an LSTM model, which is a very powerful RNN.

model = Sequential()
model.add(LSTM(128, input_shape=(WORD_LENGTH, len(unique_words))))
model.add(Dense(len(unique_words)))
model.add(Activation('softmax'))

Training the next word prediction model

I will train the next word prediction model for 20 epochs:

optimizer = RMSprop(lr=0.01)
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
history = model.fit(X, Y, validation_split=0.05, batch_size=128, epochs=2, shuffle=True).history

We have now trained our model successfully. Before going on to evaluate it, it would be better to save this model for future use.

model.save('keras_next_word_model.h5')
pickle.dump(history, open("history.p", "wb"))
model = load_model('keras_next_word_model.h5')
history = pickle.load(open("history.p", "rb"))

Evaluating the next word prediction model

Now let's take a quick look at how our model behaves based on the changes in its accuracy and loss during training:

plt.plot(history['acc'])
plt.plot(history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')

plt.plot(history['loss'])
plt.plot(history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')

Testing the next word prediction model

Now let's build a Python program to predict the next word with our trained model. For this, I will define some basic functions that will be used in the process.

def prepare_input(text):
    x = np.zeros((1, SEQUENCE_LENGTH, len(chars)))
    for t, char in enumerate(text):
        x[0, t, char_indices[char]] = 1.
        
    return x

Now, before going further, let's test the function; make sure to use the lower() function when giving the input:

prepare_input("This is an example of input for our LSTM".lower())

Note that the sequences must consist of 40 characters (not words) so that we can easily put them in a tensor of shape (57, 40, 1). Before going further, let's check whether the function we created works correctly.

def prepare_input(text):
    x = np.zeros((1, WORD_LENGTH, len(unique_words)))
    for t, word in enumerate(text.split()):
        print(word)
        x[0, t, unique_word_index[word]] = 1
    return x
prepare_input("It is not a lack".lower())

array([[[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
...,
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.]]])

Now I will create a function to return samples:

def sample(preds, top_n=3):
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds)
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)

    return heapq.nlargest(top_n, range(len(preds)), preds.take)

And now I will create a function to predict the next word:

def predict_completion(text):
    original_text = text
    generated = text
    completion = ''
    while True:
        x = prepare_input(text)
        preds = model.predict(x, verbose=0)[0]
        next_index = sample(preds, top_n=1)[0]
        next_char = indices_char[next_index]
        text = text[1:] + next_char
        completion += next_char
        
        if len(original_text + completion) + 2 > len(original_text) and next_char == ' ':
            return completion

This function was created to predict the next word until a space is generated. It does this by repeating the input, querying our RNN model and extracting instances from it. Now I will modify the function above to predict multiple characters:

def predict_completions(text, n=3):
    x = prepare_input(text)
    preds = model.predict(x, verbose=0)[0]
    next_indices = sample(preds, n)
    return [indices_char[idx] + predict_completion(text[1:] + indices_char[idx]) for idx in next_indices]

Now I will use the 40-character sequence that we can use as the basis for our predictions.

quotes =[
    "It is not a lack of love, but a lack of friendship that makes unhappy marriages.",
    "That which does not kill us makes us stronger.",
    "I'm not upset that you lied to me, I'm upset that from now on I can't believe you.",
    "And those who were seen dancing were thought to be insane by those who could not hear the music.",
    "It is hard enough to remember my opinions, without also remembering my reasons for them!"
]

Now, finally, we can use the model to predict the next word:

for q in quotes:
    seq = q[:40].lower()
    print(seq)
    print(predict_completions(seq, 5))
    print()

it is not a lack of love, but a lack of
['the ', 'an ', 'such ', 'man ', 'present, ']
that which does not kill us makes us str
['ength ', 'uggle ', 'ong ', 'ange ', 'ive ']
i'm not upset that you lied to me, i'm u
['nder ', 'pon ', 'ses ', 't ', 'uder ']
and those who were seen dancing were tho
['se ', 're ', 'ugh ', ' servated ', 't ']it is hard enough to remember my opinion
[' of ', 's ', ', ', 'nof ', 'ed ']

I hope you liked this article on the next word prediction model.

Post links

Leave a comment

Your email address will not be published. Required fields are marked *