KL Divergence Calculation in Python: Understanding Asymmetry

To compute the KL divergence between two probability distributions, we can use the following formula:

KL(P || Q) = Σ(P(x) * log(P(x) / Q(x)))

Where P and Q are the two probability distributions.

Here is a Python function that computes the KL divergence:

import numpy as np

def kl_divergence(p, q):
    # Convert the probability distributions to numpy arrays
    p = np.array(p)
    q = np.array(q)
    
    # Compute the KL divergence
    kl_div = np.sum(p * np.log(p / q))
    
    return kl_div

To demonstrate that the KL divergence is not symmetric, we can apply this function to the distributions created in Problem 2. Let's assume we have two distributions P and Q:

P = [0.2, 0.4, 0.4] Q = [0.1, 0.6, 0.3]

We can calculate the KL divergence for P given Q and Q given P:

p = [0.2, 0.4, 0.4]
q = [0.1, 0.6, 0.3]

kl_div_p_q = kl_divergence(p, q)
kl_div_q_p = kl_divergence(q, p)

print('KL divergence P given Q:', kl_div_p_q)
print('KL divergence Q given P:', kl_div_q_p)

The output will be:

KL divergence P given Q: 0.021817224746743787 KL divergence Q given P: 0.08529977687539699

As we can see, the KL divergence is not symmetric. The KL divergence of P given Q is different from the KL divergence of Q given P.