Linear Regression with LSE and BGD: Python Implementation and Analysis

This guide demonstrates how to solve a linear regression problem using Least Square Estimation (LSE) and Batch Gradient Descent (BGD) in Python. We'll use the following dataset as an example:

Training Data:

{[1.2, 0.3], 1.5}, {[-2.5, 1.1], -0.4}, {[3.4, 2.7], 2.1}, {[2.9, 6.3], -1.6}

Objective:

1. Implement LSE and BGD to learn a linear regression model without using polynomial terms.2. Predict the value of 'y' when 'x' is [1, 1].3. Analyze the performance of BGD with different maximum iteration values (Pmax = 100, 1000, 10000) by examining the Mean Squared Error (MSE).4. Visualize the convergence of BGD by plotting MSE against the number of iterations.

**Python Code:**pythonimport numpy as npimport matplotlib.pyplot as plt

Training dataX = np.array([[1.2, 0.3], [-2.5, 1.1], [3.4, 2.7], [2.9, 6.3]])y = np.array([1.5, -0.4, 2.1, -1.6])

Add a column of ones for the intercept termX_with_intercept = np.column_stack((np.ones(len(X)), X))

Least Square Estimation (LSE)w_lse = np.linalg.inv(X_with_intercept.T @ X_with_intercept) @ X_with_intercept.T @ y

Batch Gradient Descent (BGD)def compute_cost(X, y, w): m = len(y) y_pred = X @ w cost = np.sum((y_pred - y) ** 2) / (2 * m) return cost

def gradient_descent(X, y, w, learning_rate, num_iterations): m = len(y) cost_history = [] for i in range(num_iterations): y_pred = X @ w error = y_pred - y gradient = X.T @ error / m w -= learning_rate * gradient cost = compute_cost(X, y, w) cost_history.append(cost) return w, cost_history

BGD with different Pmax valuesPmax_values = [100, 1000, 10000]mse_values = []

for Pmax in Pmax_values: w_bgd = np.zeros(X_with_intercept.shape[1]) learning_rate = 0.01 num_iterations = Pmax w_bgd, cost_history = gradient_descent(X_with_intercept, y, w_bgd, learning_rate, num_iterations) mse_values.append(compute_cost(X_with_intercept, y, w_bgd))

# Plot MSE vs. Iterations for each Pmax    plt.figure(figsize=(10, 6))    plt.plot(range(1, Pmax+1), cost_history, label=f'Pmax = {Pmax}')    plt.xlabel('# Iterations')    plt.ylabel('MSE')    plt.title(f'MSE vs. # Iterations for Pmax = {Pmax}')    plt.legend()    plt.show()

Predict the value when x = [1, 1]x_pred = np.array([1, 1])x_pred_with_intercept = np.insert(x_pred, 0, 1)y_pred_lse = np.dot(w_lse, x_pred_with_intercept)y_pred_bgd = np.dot(w_bgd, x_pred_with_intercept)

print('Predicted value using LSE for x = [1, 1]:', y_pred_lse)print('Predicted value using BGD for x = [1, 1]:', y_pred_bgd)

Explanation:

1. Import Libraries: We begin by importing NumPy for numerical computations and Matplotlib for plotting.2. Prepare Data: The training data is defined, and a column of ones is added to the input features (X) to represent the intercept term.3. LSE Implementation: The LSE solution is calculated directly using matrix operations.4. BGD Implementation: We define functions for computing the cost (MSE) and implementing the BGD algorithm. 5. Iterate over Pmax Values: The code iterates through different Pmax values, performing BGD and storing the MSE for analysis.6. Prediction: We use the learned model parameters to predict the 'y' value for a new input 'x' = [1, 1].7. Visualization: The code generates plots of MSE vs. the number of iterations for each Pmax value, allowing us to visualize the convergence behavior of BGD.

This comprehensive guide provides a clear and structured approach to understanding and implementing linear regression with LSE and BGD in Python. By running this code, you will gain insights into the performance of these methods and be able to apply them to your own regression tasks.