PROBLEM 1 – Representing English Text 5 pts Read in these two GLUE datasets see section DATA above Also convert alphabetical characters to lower caseDataset Use file NotesSST SSTtraintsv Use column se
To solve this problem, we need the NLTK package installed. If you don't have it installed, you can install it by running the following command:
!pip install nltk
Once installed, we can proceed with the solution:
import nltk
from nltk.tokenize import word_tokenize
import pandas as pd
# Read in SST dataset
sst_df = pd.read_csv("SST/train.tsv", sep="\t")
sst_sentences = sst_df["sentence"].tolist()
# Convert alphabetical characters to lower case
sst_sentences = [sentence.lower() for sentence in sst_sentences]
# Tokenize sentences
sst_tokens = [word_tokenize(sentence) for sentence in sst_sentences]
# Print the first 10 tokens of SST dataset
for tokens in sst_tokens[:10]:
print(tokens)
# Read in QNLI dataset
qnli_df = pd.read_csv("QNLI/dev.tsv", sep="\t")
qnli_sentences = qnli_df["sentence"].tolist()
# Convert alphabetical characters to lower case
qnli_sentences = [sentence.lower() for sentence in qnli_sentences]
# Tokenize sentences
qnli_tokens = [word_tokenize(sentence) for sentence in qnli_sentences]
# Print the first 10 tokens of QNLI dataset
for tokens in qnli_tokens[:10]:
print(tokens)
Make sure to replace "SST/train.tsv" and "QNLI/dev.tsv" with the correct file paths to the datasets on your system.
This solution reads in the SST and QNLI datasets using the read_csv() function from the pandas library. It then extracts the "sentence" column from each dataset and converts the alphabetical characters to lower case using a list comprehension. The word_tokenize() function from the NLTK package is then applied to each sentence to tokenize it into a list of tokens. Finally, the first 10 tokens from each dataset are printed.
原文地址: https://www.cveoy.top/t/topic/i583 著作权归作者所有。请勿转载和采集!