Sentiment Analysis with Language Models: A Step-by-Step Guide

This document outlines a comprehensive guide to sentiment analysis using language models, focusing on the implementation of key concepts and their application in predicting sentiment. The guide will walk you through each step, from data preparation to model evaluation.

Problem 1: Reading the Data (5 pts)

Read the file 'train.tsv' from the Stanford Sentiment Treebank (SST) as shared in the GLUE task.
Split the dataset into training, testing, and validation sets with these sizes:
- Validation: 100 rows
- Test: 100 rows
- Training: All remaining rows.
Analyze the 'label' column, indicating positive (1) or negative (0) sentiment. Calculate the prior probability of each class in the training set.

Problem 2: Tokenizing Data (10 pts)

Create a function that takes a sentence as input and converts it to a tokenized sequence padded by start ('~~') and end ('~~') symbols. Example: 'hello class' -> ['~~', 'hello', 'class', '~~'].
Apply the function to all sentences in the training set. Show the tokenization of the first training sentence.
Determine the vocabulary size of the training set, including the start and end symbols.

Problem 3: Bigram Counts (10 pts)

Write a function to count bigram frequencies in a list of tokenized sequences. Return a dictionary of dictionaries where bigram_counts['wi']['wj'] gives the frequency of the bigram ('wi', 'wj').
Apply the function to the output of Problem 2 to build a counter for all training sentences.
Use the result to find the frequency of sentences starting with 'the' (i.e., the bigram ('~~', 'the')).~~

Problem 4: Smoothing (20 pts)

Implement the smoothing formula [6.13] from the E-NLP textbook (page 129, 6.2 Smoothing and discounting). Write a function to calculate the (negative) log-probability of a word given the previous word, using:

The current word (wm)

The previous word (wm-1)

Bigram counts (output of Problem 3)

Alpha, a smoothing parameter

Vocabulary size

Use this function to show the log probability of 'academy' following 'award' with alpha=0.001 and alpha=0.5. You should observe significantly different results.

Problem 5: Sentence Log-Probability (10 pts)

Create a function to return the log-probability of a sentence, assuming a word's probability depends only on the previous word. Use:

The sentence as a Python string

Bigram counts (output of Problem 3)

Alpha, a smoothing parameter

Vocabulary size

Calculate the log probabilities of these sentences:

'this was a really great movie but it was a little too long.'

'long too little a was it but movie great really a was this.'

Problem 6: Tuning Alpha (10 pts)

Utilize the validation set to find an optimal 'alpha' value.

Apply the function from Problem 5 to the validation set using three different 'alpha' values (e.g., 0.001, 0.01, 0.1). Show the log-likelihood estimate of the validation set for each value.

Select the 'alpha' value yielding the best result and store it in a variable named 'selected_alpha'.

Problem 7: Applying Language Models (20 pts)

Classify the test set of 100 sentences by sentiment using your previous work and modeling the language of both positive and negative sentiments. Follow these steps:

Separate the training dataset into positive and negative sentences. Compute vocabulary size and bigram counts for each set.

For each test sentence:

Compute 'positive sentiment score' and 'negative sentiment score' using:

The function from Problem 5

Bayes' Rule

Class priors calculated in Problem 1

Assign a predicted sentiment label based on the comparison of scores.

Determine the class distribution of your predicted labels (how often was positive sentiment predicted, correctly or incorrectly? How often was negative sentiment predicted?).

Compare predicted labels to true sentiment labels and calculate the accuracy of the experiment.

You can reuse the 'selected_alpha' value from Problem 6; re-tuning is optional.

Problem 8: Markov Assumption (10 pts - Answer in Blackboard)

Identify where the Markov assumption was applied in this homework.

Imagine applying the 2nd-order Markov assumption using trigrams. Would you expect the accuracy results to increase or decrease? Provide a reason or a benefit/drawback of using trigrams for this experiment.

This guide provides a step-by-step approach to implementing sentiment analysis using language models, emphasizing the importance of data preparation, tokenization, bigram counts, smoothing, and parameter tuning. This foundation can be extended to more sophisticated models and techniques for improved accuracy and performance.