LSTM in Machine Learning

The LSTM Network model stands for Long Short Term Memory networks. These are a special kind of Neural Networks which are generally capable of understanding long term dependencies. LSTM model was generally designed to prevent the problems of long term dependencies which they generally do in a very good manner. In this article I will take you through how we can use LSTMs in Time Series Forecasting.

The LSTM Network models generally have potential to remove or add data carefully which is regulated by a special structure known as gates. The first step in processing LSTMs is to determine what information we need to throw from the cell.

The next step is deciding what new information we should store in the cell. At last we decide what we want as an output. The output is generally based on the state of cells. Let’s understand the process of LSTM using an example of Time Series Forecasting to predict stock prices.

Stock Price Prediction using LSTM

Let’s see how we can use the LSTM model to predict stock prices using Time Series Forecasting. For this task I will scrape the data from yahoo finance using the pandas_datareader library. So before doing so let’s start with importing all the packages we need for this task:

import math
import matplotlib.pyplot as plt
import keras
import pandas as pd
import numpy as np
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers import Dropout
from keras.layers import *
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from keras.callbacks import EarlyStopping
import pandas_datareader as webCode language: Python (python)

To get the data:

data = web.DataReader("", data_source="yahoo", start=None, end=None)
data.reset_index(inplace=True)
data.head()Code language: Python (python)

The next step is to divide the data into training and testing sets to avoid overfitting and to be able to study the generalizability of our model:

training_set = data.iloc[:800, 1:2].values
test_set = data.iloc[800:, 1:2].valuesCode language: Python (python)

The target value to be predicted will be the value of the “Close” share price. It is a good idea to normalize the data before fitting the model. This will increase performance. Let’s create the input features with a 1-day lag:

# Feature Scaling
sc = MinMaxScaler(feature_range = (0, 1))
training_set_scaled = sc.fit_transform(training_set)
# Creating a data structure with 60 time-steps and 1 output
X_train = []
y_train = []
for i in range(60, 800):
  X_train.append(training_set_scaled[i-60:i, 0])
  y_train.append(training_set_scaled[i, 0])
X_train, y_train = np.array(X_train), np.array(y_train)
X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 1))Code language: Python (python)

Building LSTM Model

Now it’s time to build the Long-Short Term Memory model, I will build the neural network with 50 neurons and four hidden layers:

model = Sequential()
#Adding the first LSTM layer and some Dropout regularisation
model.add(LSTM(units = 50, return_sequences = True, input_shape = (X_train.shape[1], 1)))
model.add(Dropout(0.2))
# Adding a second LSTM layer and some Dropout regularisation
model.add(LSTM(units = 50, return_sequences = True))
model.add(Dropout(0.2))
# Adding a third LSTM layer and some Dropout regularisation
model.add(LSTM(units = 50, return_sequences = True))
model.add(Dropout(0.2))
# Adding a fourth LSTM layer and some Dropout regularisation
model.add(LSTM(units = 50))
model.add(Dropout(0.2))
# Adding the output layer
model.add(Dense(units = 1))

# Compiling the RNN
model.compile(optimizer = 'adam', loss = 'mean_squared_error')

# Fitting the RNN to the Training set
model.fit(X_train, y_train, epochs = 100, batch_size = 32)Code language: Python (python)

The model will take some time to run, after the execution you will see an output like this:

Epoch 1/100 24/24 [==============================] - 2s 99ms/step - loss: 0.0496 Epoch 2/100 24/24 [==============================] - 3s 111ms/step - loss: 0.0085 Epoch 3/100 24/24 [==============================] - 3s 109ms/step - loss: 0.0057 Epoch 4/100 24/24 [==============================] - 3s 105ms/step - loss: 0.0057 Epoch 5/100 24/24 [==============================] - 3s 111ms/step - loss: 0.0054 Epoch 6/100 24/24 [==============================] - 3s 106ms/step - loss: 0.0062 Epoch 7/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0055 Epoch 8/100 24/24 [==============================] - 3s 105ms/step - loss: 0.0050 Epoch 9/100 24/24 [==============================] - 3s 113ms/step - loss: 0.0044 Epoch 10/100 24/24 [==============================] - 3s 105ms/step - loss: 0.0053 Epoch 11/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0048 Epoch 12/100 24/24 [==============================] - 3s 118ms/step - loss: 0.0051 Epoch 13/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0054 Epoch 14/100 24/24 [==============================] - 3s 111ms/step - loss: 0.0046 Epoch 15/100 24/24 [==============================] - 3s 106ms/step - loss: 0.0046 Epoch 16/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0045 Epoch 17/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0044 Epoch 18/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0042 Epoch 19/100 24/24 [==============================] - 3s 112ms/step - loss: 0.0039 Epoch 20/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0042 Epoch 21/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0043 Epoch 22/100 24/24 [==============================] - 3s 109ms/step - loss: 0.0038 Epoch 23/100 24/24 [==============================] - 3s 113ms/step - loss: 0.0037 Epoch 24/100 24/24 [==============================] - 3s 106ms/step - loss: 0.0039 Epoch 25/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0038 Epoch 26/100 24/24 [==============================] - 3s 114ms/step - loss: 0.0040 Epoch 27/100 24/24 [==============================] - 3s 117ms/step - loss: 0.0034 Epoch 28/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0039 Epoch 29/100 24/24 [==============================] - 3s 109ms/step - loss: 0.0035 Epoch 30/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0055 Epoch 31/100 24/24 [==============================] - 3s 106ms/step - loss: 0.0038 Epoch 32/100 24/24 [==============================] - 3s 113ms/step - loss: 0.0033 Epoch 33/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0035 Epoch 34/100 24/24 [==============================] - 3s 111ms/step - loss: 0.0033 Epoch 35/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0035 Epoch 36/100 24/24 [==============================] - 3s 114ms/step - loss: 0.0036 Epoch 37/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0033 Epoch 38/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0039 Epoch 39/100 24/24 [==============================] - 3s 109ms/step - loss: 0.0033 Epoch 40/100 24/24 [==============================] - 3s 109ms/step - loss: 0.0032 Epoch 41/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0031 Epoch 42/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0028 Epoch 43/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0038 Epoch 44/100 24/24 [==============================] - 3s 106ms/step - loss: 0.0034 Epoch 45/100 24/24 [==============================] - 3s 106ms/step - loss: 0.0034 Epoch 46/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0032 Epoch 47/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0033 Epoch 48/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0046 Epoch 49/100 24/24 [==============================] - 3s 115ms/step - loss: 0.0032 Epoch 50/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0028 Epoch 51/100 24/24 [==============================] - 3s 106ms/step - loss: 0.0028 Epoch 52/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0025 Epoch 53/100 24/24 [==============================] - 3s 105ms/step - loss: 0.0025 Epoch 54/100 24/24 [==============================] - 3s 105ms/step - loss: 0.0027 Epoch 55/100 24/24 [==============================] - 3s 116ms/step - loss: 0.0033 Epoch 56/100 24/24 [==============================] - 3s 109ms/step - loss: 0.0029 Epoch 57/100 24/24 [==============================] - 3s 112ms/step - loss: 0.0026 Epoch 58/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0026 Epoch 59/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0029 Epoch 60/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0028 Epoch 61/100 24/24 [==============================] - 3s 106ms/step - loss: 0.0026 Epoch 62/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0030 Epoch 63/100 24/24 [==============================] - 3s 105ms/step - loss: 0.0027 Epoch 64/100 24/24 [==============================] - 3s 111ms/step - loss: 0.0030 Epoch 65/100 24/24 [==============================] - 3s 111ms/step - loss: 0.0024 Epoch 66/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0028 Epoch 67/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0024 Epoch 68/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0022 Epoch 69/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0023 Epoch 70/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0024 Epoch 71/100 24/24 [==============================] - 3s 115ms/step - loss: 0.0022 Epoch 72/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0022 Epoch 73/100 24/24 [==============================] - 3s 105ms/step - loss: 0.0025 Epoch 74/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0020 Epoch 75/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0021 Epoch 76/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0022 Epoch 77/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0021 Epoch 78/100 24/24 [==============================] - 3s 112ms/step - loss: 0.0022 Epoch 79/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0020 Epoch 80/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0020 Epoch 81/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0021 Epoch 82/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0021 Epoch 83/100 24/24 [==============================] - 3s 111ms/step - loss: 0.0019 Epoch 84/100 24/24 [==============================] - 3s 110ms/step - loss: 0.0020 Epoch 85/100 24/24 [==============================] - 3s 105ms/step - loss: 0.0019 Epoch 86/100 24/24 [==============================] - 3s 111ms/step - loss: 0.0018 Epoch 87/100 24/24 [==============================] - 3s 109ms/step - loss: 0.0018 Epoch 88/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0020 Epoch 89/100 24/24 [==============================] - 3s 111ms/step - loss: 0.0027 Epoch 90/100 24/24 [==============================] - 3s 109ms/step - loss: 0.0024 Epoch 91/100 24/24 [==============================] - 3s 111ms/step - loss: 0.0021 Epoch 92/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0016 Epoch 93/100 24/24 [==============================] - 3s 106ms/step - loss: 0.0017 Epoch 94/100 24/24 [==============================] - 3s 111ms/step - loss: 0.0018 Epoch 95/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0019 Epoch 96/100 24/24 [==============================] - 3s 109ms/step - loss: 0.0018 Epoch 97/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0020 Epoch 98/100 24/24 [==============================] - 3s 107ms/step - loss: 0.0017 Epoch 99/100 24/24 [==============================] - 3s 108ms/step - loss: 0.0019 Epoch 100/100 24/24 [==============================] - 3s 111ms/step - loss: 0.0018
<tensorflow.python.keras.callbacks.History at 0x7ffa67b17b00>

Now, let’s reshape the test data:

dataset_train = data.iloc[:800, 1:2]
dataset_test = data.iloc[800:, 1:2]
dataset_total = pd.concat((dataset_train, dataset_test), axis = 0)
inputs = dataset_total[len(dataset_total) - len(dataset_test) - 60:].values
inputs = inputs.reshape(-1,1)
inputs = sc.transform(inputs)
X_test = []
for i in range(60, 519):
  X_test.append(inputs[i-60:i, 0])
X_test = np.array(X_test)
X_test = np.reshape(X_test, (X_test.shape[0], X_test.shape[1], 1))Code language: Python (python)

Now, let’s make predictions using the LSTM model on the test set:

predicted_stock_price = model.predict(X_test)
predicted_stock_price = sc.inverse_transform(predicted_stock_price)Code language: Python (python)

Now, let’s have a look at our predictions:

plt.plot(df.loc[800:, 'Date'],dataset_test.values, color = 'red', label = 'Real TESLA Stock Price')
plt.plot(df.loc[800:, 'Date'],predicted_stock_price, color = 'blue', label = 'Predicted TESLA Stock Price')
plt.xticks(np.arange(0,459,50))
plt.title('TESLA Stock Price Prediction')
plt.xlabel('Time')
plt.ylabel('TESLA Stock Price')
plt.legend()
plt.show()Code language: Python (python)

Also, Read – What is Data Mining?

We can see that our model has worked very well. It can accurately track most unanticipated jumps/declines; however, for the most recent timestamps, we can see that the model expects lower (predicted) values ​​compared to the actual price values ​​of the action.

I hope you liked this article on LSTM in Machine Learning. Feel free to ask your valuable questions in the comments section below. You can also follow me on Medium to learn every topic of Machine Learning.

Also, Read – Scrape Wikipedia with Python.

Follow Us: