RNN Classifier Model for Text Classification with Improved Accuracy

Improving RNN Classifier Accuracy for Text Classification

This article provides a step-by-step guide to improving the accuracy of an RNN Classifier model for text classification tasks. We'll explore several techniques that can be applied to enhance the model's performance.

The Original Model:

class RNNClassifier(nn.Module):  
    def __init__(self, vocab_size, embedding_dim, hidden_dim, label_size, padding_idx):  
        super(RNNClassifier, self).__init__()  
        self.vocab_size = vocab_size  
        self.embedding_dim = embedding_dim  
        self.hidden_dim = hidden_dim  
        self.label_size = label_size  
        self.num_layers = 2
  
        # Embedding Layer  
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=padding_idx)  
          
        # RNN Layer  
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, num_layers=self.num_layers, batch_first=True)  
          
        # Output Layer  
        self.fc = nn.Linear(hidden_dim, label_size)  
  
    def zero_state(self, batch_size):  
        hidden = torch.zeros(self.num_layers,batch_size,  self.hidden_dim)  
        cell = torch.zeros(self.num_layers,batch_size,  self.hidden_dim) 
        return hidden, cell  
  
    def forward(self, text):  
        # text shape = [batch_size, seq_len]  
        # Embedding  
        emb = self.embedding(text)  # shape = [batch_size, seq_len, embedding_dim]  
          
        # LSTM Layer  
        h0, c0 = self.zero_state(text.size(0))  # shape = [batch_size, hidden_dim]  
        output, (hn, cn) = self.lstm(emb, (h0, c0))  # output shape = [batch_size, seq_len, hidden_dim], hn shape = [batch_size, num_layers, hidden_dim], cn shape = [batch_size, num_layers, hidden_dim]  
          
        # Output Layer  
        output = self.fc(output[:, -1, :])  # use the last output at the last step for classification  
        return output

Improving the Model:

Given a Test Loss of 1.007 and Test Accuracy of 58.43%, we can explore several strategies to enhance the model's performance:

1. Increase Hidden Layers: Adding more hidden layers provides greater complexity and representational capacity, enabling the model to capture more intricate patterns in the input sequences.

2. Increase Hidden Layer Size: Expanding the dimensionality of hidden layers increases the model's capacity, potentially leading to better representation and classification.

3. Use Bi-directional LSTM: Employing a bi-directional LSTM allows the model to consider both forward and backward information from the input sequence, providing a more holistic understanding of the text and improving accuracy.

4. Add Dropout: Including a dropout layer after the LSTM layer helps reduce overfitting, improving the model's generalization ability and preventing it from memorizing training data.

5. Utilize Pre-trained Word Embeddings: Leveraging pre-trained word embeddings as the initialization for the embedding layer allows the model to benefit from pre-existing semantic relationships between words, enhancing its ability to capture meaningful representations.

Modified Model Code:

class RNNClassifier(nn.Module):  
    def __init__(self, vocab_size, embedding_dim, hidden_dim, label_size, padding_idx):  
        super(RNNClassifier, self).__init__()  
        self.vocab_size = vocab_size  
        self.embedding_dim = embedding_dim  
        self.hidden_dim = hidden_dim  
        self.label_size = label_size  
        self.num_layers = 2
  
        # Embedding Layer  
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=padding_idx)  
          
        # RNN Layer  
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, num_layers=self.num_layers, batch_first=True, bidirectional=True)  
          
        # Output Layer  
        self.fc = nn.Linear(hidden_dim*2, label_size)  # double the hidden_dim due to bidirectional LSTM
  
        # Dropout Layer
        self.dropout = nn.Dropout(0.5)
  
    def zero_state(self, batch_size):  
        hidden = torch.zeros(self.num_layers*2, batch_size, self.hidden_dim)  # double the num_layers due to bidirectional LSTM
        cell = torch.zeros(self.num_layers*2, batch_size, self.hidden_dim)
        return hidden, cell  
  
    def forward(self, text):  
        # text shape = [batch_size, seq_len]  
        # Embedding  
        emb = self.embedding(text)  # shape = [batch_size, seq_len, embedding_dim]  
          
        # LSTM Layer  
        h0, c0 = self.zero_state(text.size(0))  # shape = [batch_size, hidden_dim]  
        output, (hn, cn) = self.lstm(emb, (h0, c0))  # output shape = [batch_size, seq_len, hidden_dim*2], hn shape = [batch_size, num_layers*2, hidden_dim], cn shape = [batch_size, num_layers*2, hidden_dim]  
          
        # Output Layer  
        output = self.dropout(output[:, -1, :])  # use the last output at the last step for classification
        output = self.fc(output)  
        return output

Conclusion:

This modified model incorporates the suggested improvements, specifically adding a bi-directional LSTM layer, a dropout layer, and adjusting the hidden layer dimensions. These adjustments contribute to a more robust model, potentially leading to improved accuracy. However, remember that optimal hyperparameter values and model architecture depend on the specific dataset and task. You can fine-tune these parameters to further enhance performance.