NLP使用SST-2数据集实例通过应用NLTK中的函数word tokenize 0将每个数据集转换为单个令牌列表 计算Word probability

首先，我们需要导入必要的库和数据集：

import nltk
from nltk.tokenize import word_tokenize
from nltk.probability import FreqDist
from nltk.corpus import stopwords

nltk.download('punkt')
nltk.download('stopwords')

# 读取数据集
with open('SST-2/train.txt', 'r') as f:
    train_data = f.readlines()

with open('SST-2/dev.txt', 'r') as f:
    dev_data = f.readlines()

with open('SST-2/test.txt', 'r') as f:
    test_data = f.readlines()

# 停用词
stop_words = set(stopwords.words('english'))

# 定义函数进行令牌化和去除停用词
def tokenize_and_remove_stopwords(text):
    tokens = word_tokenize(text.lower())
    tokens = [token for token in tokens if token.isalpha() and token not in stop_words]
    return tokens

接下来，我们可以使用上述函数将数据集中的每个文本转换为令牌列表，并计算每个令牌的词频分布：

# 计算训练集中的词频分布
train_tokens = [tokenize_and_remove_stopwords(text) for text in train_data]
train_tokens = [token for sublist in train_tokens for token in sublist]
train_freq_dist = FreqDist(train_tokens)

# 计算开发集中的词频分布
dev_tokens = [tokenize_and_remove_stopwords(text) for text in dev_data]
dev_tokens = [token for sublist in dev_tokens for token in sublist]
dev_freq_dist = FreqDist(dev_tokens)

# 计算测试集中的词频分布
test_tokens = [tokenize_and_remove_stopwords(text) for text in test_data]
test_tokens = [token for sublist in test_tokens for token in sublist]
test_freq_dist = FreqDist(test_tokens)

最后，我们可以使用词频分布来计算每个令牌的概率：

# 计算训练集中每个令牌的概率
train_word_prob = {}
total_train_tokens = len(train_tokens)
for token in train_freq_dist:
    train_word_prob[token] = train_freq_dist[token] / total_train_tokens

# 计算开发集中每个令牌的概率
dev_word_prob = {}
total_dev_tokens = len(dev_tokens)
for token in dev_freq_dist:
    dev_word_prob[token] = dev_freq_dist[token] / total_dev_tokens

# 计算测试集中每个令牌的概率
test_word_prob = {}
total_test_tokens = len(test_tokens)
for token in test_freq_dist:
    test_word_prob[token] = test_freq_dist[token] / total_test_tokens

现在，我们分别得到了训练集、开发集和测试集中每个令牌的概率。您可以根据需要使用这些概率进行后续的NLP任务