Python Function for Creating Probability Distributions from Token Lists - Problem 2

Here is an example of a Python function that creates a probability distribution from a list of tokens:\n\npython\ndef create_probability_distribution(tokens):\n total_tokens = len(tokens)\n distribution = {}\n \n for token in tokens:\n if token in distribution:\n distribution[token] += 1\n else:\n distribution[token] = 1\n \n for token, count in distribution.items():\n distribution[token] = count / total_tokens\n \n return distribution\n\n\nNow, let's apply this function to the lists created in Problem 1 for SST and QNLI:\n\npython\nsst_distribution = create_probability_distribution(sst_tokens)\nqnli_distribution = create_probability_distribution(qnli_tokens)\n\n\nTo check if the probability distributions sum to 1, we can calculate the sum of all the probabilities in each distribution and compare it to 1:\n\npython\nsst_sum = sum(sst_distribution.values())\nqnli_sum = sum(qnli_distribution.values())\n\nprint("SST Distribution Sum:", sst_sum)\nprint("QNLI Distribution Sum:", qnli_sum)\n\n\nIf both sums are close to 1 (allowing for some small numerical rounding error), then we can conclude that the probability distributions are valid. If they are not close to 1, it could be due to errors in the function or other factors that need to be investigated.