Python Adaboost with KNN Base Classifier: A Practical Guide

This code demonstrates how to use a KNN model as the base classifier within an Adaboost algorithm in Python.

from sklearn.ensemble import AdaBoostClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Generate a binary classification dataset
X, y = make_classification(n_samples=1000, n_features=10, n_informative=5,
                            n_redundant=0, n_classes=2, random_state=0)

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

# Create a KNN classifier
knn = KNeighborsClassifier(n_neighbors=3)

# Create an AdaBoost classifier with KNN as the base estimator
ada = AdaBoostClassifier(base_estimator=knn, n_estimators=10, learning_rate=1)

# Train the AdaBoost classifier
ada.fit(X_train, y_train)

# Evaluate the classifier's performance on the test set
accuracy = ada.score(X_test, y_test)
print('AdaBoost with KNN base classifier accuracy:', accuracy)

Code Breakdown:

1. Data Generation: The make_classification function generates a synthetic dataset suitable for binary classification.
2. Data Splitting: The dataset is divided into training and testing sets for model training and evaluation, respectively.
3. KNN Classifier: A KNN classifier is initialized with n_neighbors set to 3, determining the number of nearest neighbors to consider for predictions.
4. AdaBoost with KNN: The AdaBoostClassifier is created, using the KNN classifier as the base_estimator. The number of estimators (n_estimators) is set to 10, and the learning_rate is 1.
5. Training: The AdaBoost classifier is trained on the training data using the fit method.
6. Evaluation: The trained AdaBoost classifier's accuracy is measured on the test set using the score method, providing a performance metric for the combined model.

This code provides a clear example of using a KNN model within an AdaBoost framework. It demonstrates the flexibility of ensemble methods and how they can leverage the strengths of different base classifiers.