出5道关于Spark相关的几道程序设计题并给出python答案

编写一个程序，使用Spark计算一个文本文件中每个单词出现的次数，并按照出现次数从大到小排序输出前10个单词及其出现次数。

from pyspark import SparkContext

sc = SparkContext("local", "wordcount")
text_file = sc.textFile("file.txt")
word_counts = text_file.flatMap(lambda line: line.split(" ")) \
    .map(lambda word: (word, 1)) \
    .reduceByKey(lambda a, b: a + b) \
    .map(lambda x: (x[1], x[0])) \
    .sortByKey(False)
top_10 = word_counts.take(10)
for count, word in top_10:
    print("{}: {}".format(word, count))

编写一个程序，使用Spark计算一个文本文件中每个单词的平均长度，并输出结果。

from pyspark import SparkContext

sc = SparkContext("local", "avgwordlen")
text_file = sc.textFile("file.txt")
word_lengths = text_file.flatMap(lambda line: line.split(" ")) \
    .map(lambda word: len(word)) \
    .filter(lambda length: length > 0)
avg_len = word_lengths.mean()
print("Average word length: {}".format(avg_len))

编写一个程序，使用Spark计算一个文本文件中每个单词的出现频率，并输出结果。

from pyspark import SparkContext

sc = SparkContext("local", "wordfreq")
text_file = sc.textFile("file.txt")
word_counts = text_file.flatMap(lambda line: line.split(" ")) \
    .map(lambda word: (word, 1)) \
    .reduceByKey(lambda a, b: a + b)
total_count = word_counts.map(lambda x: x[1]).sum()
freqs = word_counts.map(lambda x: (x[0], x[1] / total_count))
for word, freq in freqs.collect():
    print("{}: {}".format(word, freq))

编写一个程序，使用Spark计算一个文本文件中每个单词的出现次数，并将结果保存到HDFS中。

from pyspark import SparkContext

sc = SparkContext("local", "wordcount")
text_file = sc.textFile("file.txt")
word_counts = text_file.flatMap(lambda line: line.split(" ")) \
    .map(lambda word: (word, 1)) \
    .reduceByKey(lambda a, b: a + b)
word_counts.saveAsTextFile("hdfs://localhost:9000/wordcounts")

编写一个程序，使用Spark计算一个文本文件中每个单词的出现次数，并将结果保存到MySQL数据库中。

from pyspark import SparkContext
from pyspark.sql import SparkSession

sc = SparkContext("local", "wordcount")
spark = SparkSession.builder.appName("wordcount").getOrCreate()
text_file = sc.textFile("file.txt")
word_counts = text_file.flatMap(lambda line: line.split(" ")) \
    .map(lambda word: (word, 1)) \
    .reduceByKey(lambda a, b: a + b)
word_counts_df = word_counts.toDF(["word", "count"])
word_counts_df.write \
    .format("jdbc") \
    .option("url", "jdbc:mysql://localhost:3306/mydatabase") \
    .option("driver", "com.mysql.jdbc.Driver") \
    .option("dbtable", "word_counts") \
    .option("user", "myusername") \
    .option("password", "mypassword") \
    .mode("overwrite") \
    .save()
``