Recurrent Neural Networks

1. Overview

In this tutorial, we’ll talk about Recurrent Neural Networks (RNNs), which are one of the most successful architectures for processing sequential data. First, we’ll describe their basic architecture and differences from traditional feedforward neural networks. Then, we’ll present some types of RNNs, and finally, we’ll move into their applications.

2. Basic Architecture

First, let’s discuss the basic architecture of an RNN to get a deeper knowledge of why these networks perform so well on sequential data.

Every RNN consists of a series of repeating modules that are called cells and process the input data sequentially. That means that the first cell takes as input the first sample of the sequence, the second cell takes the second sample, and so on. Each cell takes the input vector and, after some processing, generates a vector known as a hidden state that is passed to the next cell . That means that each time the hidden state captures all the information given so far, enabling the network to have some memory.

In the image below, we can see a basic diagram that illustrates the basic architecture of an RNN:

We can build an RNN in Python using TensorFlow or PyTorch. The following codes give an example of how to build a classifier using RNN in TensorFlow:

import tensorflow as tf

class RNNModel(tf.keras.Model):
    def __init__(self, units, input_feature_dim, num_classes):
        super(RNNModel, self).__init__()
        # The first RNN layer
        self.rnn_layer_1 = tf.keras.layers.SimpleRNN(units, return_sequences=True, input_shape=(None, input_feature_dim))
        # The second RNN layer
        self.rnn_layer2 = tf.keras.layers.SimpleRNN(units, return_sequences=False)
        # Full connected layer
        self.fc = tf.keras.layers.Dense(num_classes)
        
    def call(self, inputs):
        x = self.rnn_layer(inputs)
        output = self.fc(x)
        return output

# Hyperparameters
input_feature_dim = 10 # Number of features
units = 20 # Number of units in RNN cell
num_classes = 3 # Number of output classes

model = RNNModel(units, input_feature_dim, num_classes)

In this example, input_shape is defined as (None, input_feature_dim), where None stands for an unspecified number of timesteps. It allows for variable sequence lengths. When there are multiple layers in the RNN, we set return_sequences=True for all the layers apart from the last one. By setting return_sequences=True, the network passes the sequence of outputs to the next layer.

We can also build the same classifier using PyTorch:

import torch
import torch.nn as nn

class RNNModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(RNNModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        # RNN layers
        self.rnn = nn.RNN(input_size, hidden_size, num_layers, batch_first=True)
        # Full connected layer
        self.fc = nn.Linear(hidden_size, num_classes)
    
    def forward(self, x):
        # Initialize hidden and cell states
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device) 
        
        # Forward propagate the RNN
        out, _ = self.rnn(x, h0)  
        
        # Decode the hidden state of the last time step
        out = self.fc(out[:, -1, :])
        return out

# Hyperparameters
input_size = 10 # Number of features for each time step in the sequence
hidden_size = 20 # Number of features in the hidden state
num_layers = 2 # Number of recurrent layers
num_classes = 3 # Number of output classes

model = RNNModel(input_size, hidden_size, num_layers, num_classes)

In PyTorch, we can build a multi-layer RNN using the parameter num_layers in nn.RNN instead of creating multiple RNN layers as we did in TensorFlow.

3. Difference with Traditional Networks

To better understand the RNN architecture, we should investigate its major differences from the traditional feedforward neural networks.

The key difference between these two architectures is that RNNs contain a continuous loop in the network that enables the input sequence to flow through the layers of the network many times. This characteristic enables RNNs to be very effective in processing sequential data, where an important part is to keep track of the ‘past’ of the sequence using some memory. The necessary memory block is represented by the hidden states that are used in the processing of the next inputs.

In the image below, we can see the two architectures that we compared where the ‘loop’ of the RNN is illustrated:

4. Types of RNNs

Throughout the years, many types of RNNs have been proposed to tackle the challenges that exist in RNNs. Now, let’s discuss the two most commonly used variations of RNNs.

4.1. LSTM

When an RNN processes very long sequences, the problem of vanishing gradients appears, meaning that the gradients of the loss function approach zero, making the network hard to train.

A Long Short-Term Memory Network (LSTM) is a variation of an RNN specifically designed to deal with the problem of vanishing gradients. It uses a memory cell that is able to maintain useful information for a long period of time without significantly decreasing the gradients of the network.

We can create an LSTM network using TensorFlow:

class LSTMModel(tf.keras.Model):
    def __init__(self, units, input_feature_dim, num_classes):
        super(LSTMModel, self).__init__()
        # LSTM layer
        self.lstm_layer = tf.keras.layers.LSTM(units, return_sequences=False, input_shape=(None, input_feature_dim))
        self.fc = tf.keras.layers.Dense(num_classes)
        
    def call(self, inputs):
        x = self.lstm_layer(inputs)
        output = self.fc(x)
        return output

And PyTorch:

import torch
import torch.nn as nn

class LSTMModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(LSTMModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        # LSTM layer
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, num_classes)
    
    def forward(self, x):
        # Initialize hidden state and cell state
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device)
        
        # Forward propagate LSTM
        out, _ = self.lstm(x, (h0, c0))  # out: tensor of shape (batch_size, seq_length, hidden_size)
        
        out = self.fc(out[:, -1, :])
        return out

4.2. GRU

Another common architecture is the Gated Recurrent Unit (GRU) which is similar to LSTMs but is much simpler in its structure and significantly faster when computing the output.

In TensorFlow, we can build a GRU network as follows:

import tensorflow as tf

class GRUModel(tf.keras.Model):
    def __init__(self, units, input_feature_dim, num_classes):
        super(GRUModel, self).__init__()
        # GRU layer
        self.gru_layer = tf.keras.layers.GRU(units, return_sequences=False, input_shape=(None, input_feature_dim))
        self.fc = tf.keras.layers.Dense(num_classes)
        
    def call(self, inputs):
        x = self.gru_layer(inputs)
        output = self.fc(x)
        return output

In PyTorch:

import torch
import torch.nn as nn

class GRUModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(GRUModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        # GRU layer
        self.gru = nn.GRU(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, num_classes)
    
    def forward(self, x):
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device)
        
        # Forward propagate GRU
        out, _ = self.gru(x, h0)  # out: tensor of shape (batch_size, seq_length, hidden_size)
        
        out = self.fc(out[:, -1, :])
        return out

5. Applications

Now that we have gained an understanding of the architectures of RNNs, we’ll talk about their most common applications of them.

5.1. Natural Language Processing

The most common application of RNNs is in processing text sequences. For example, we can use an RNN to generate text by predicting each time the next word of a text sequence. This task can be proved helpful for chatbot applications and online language translators.

5.2. Series Forecasting

Another useful application of RNNs is in time series forecasting, where our goal is to predict some future outcomes given the previous ones. For example, in weather forecasting, we can use an RNN to explore historical patterns in previous weather data and employ them to predict the weather in the future better.

5.3. Video Processing

Videos can be considered as another type of sequence since they are sequences of image frames. So, RNNs can also be used in video processing tasks like video classification, where our goal is to classify a given video into a category.

5.4. Reinforcement Learning

Finally, RNNs are also used in cases where we want to make decisions in a dynamic environment, like in reinforcement learning. Specifically, we can control a robot using an RNN that maintains the current state of the environment in the RNN memory.

6. Conclusion

In this article, we presented RNNs. First, we talked about their architecture and their differences from traditional networks. Then, we briefly discussed two types of RNNs and some of their applications.