To solve this problem, we need the NLTK package installed. If you don't have it installed, you can install it by running the following command:

!pip install nltk

Once installed, we can proceed with the solution:

import nltk
from nltk.tokenize import word_tokenize
import pandas as pd

# Read in SST dataset
sst_df = pd.read_csv("SST/train.tsv", sep="\t")
sst_sentences = sst_df["sentence"].tolist()

# Convert alphabetical characters to lower case
sst_sentences = [sentence.lower() for sentence in sst_sentences]

# Tokenize sentences
sst_tokens = [word_tokenize(sentence) for sentence in sst_sentences]

# Print the first 10 tokens of SST dataset
for tokens in sst_tokens[:10]:
    print(tokens)

# Read in QNLI dataset
qnli_df = pd.read_csv("QNLI/dev.tsv", sep="\t")
qnli_sentences = qnli_df["sentence"].tolist()

# Convert alphabetical characters to lower case
qnli_sentences = [sentence.lower() for sentence in qnli_sentences]

# Tokenize sentences
qnli_tokens = [word_tokenize(sentence) for sentence in qnli_sentences]

# Print the first 10 tokens of QNLI dataset
for tokens in qnli_tokens[:10]:
    print(tokens)

Make sure to replace "SST/train.tsv" and "QNLI/dev.tsv" with the correct file paths to the datasets on your system.

This solution reads in the SST and QNLI datasets using the read_csv() function from the pandas library. It then extracts the "sentence" column from each dataset and converts the alphabetical characters to lower case using a list comprehension. The word_tokenize() function from the NLTK package is then applied to each sentence to tokenize it into a list of tokens. Finally, the first 10 tokens from each dataset are printed.

PROBLEM 1 – Representing English Text 5 pts Read in these two GLUE datasets see section DATA above Also convert alphabetical characters to lower caseDataset Use file NotesSST SSTtraintsv Use column se

原文地址: https://www.cveoy.top/t/topic/i583 著作权归作者所有。请勿转载和采集!

免费AI点我,无需注册和登录