Supervised Classification | hyperparameter optimization and voting ensemble


 

Supervised Classification | hyperparameter optimization and voting ensemble

Using the Titanic dataset, in this exercise, we will perform GridSearchCV on five different classification algorithms to fit the models using the best-performing hyperparameters.

Afterwards, we will use an ensemble voting method, a technique in machine learning where multiple individual models are combined to make predictions. 

The idea behind ensemble methods is that by combining the predictions of multiple models, overall performance can be better than that of individual models.

In this case we will use:

Soft Voting: In this approach, each model provides a probability or confidence score for each class, and the scores are averaged or weighted to make the final decision. This is particularly useful when individual models can produce probability estimates.

Hard Voting: Each model in the ensemble predicts a class, and the class with the most votes is chosen as the final prediction.


#import necessary libraries

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import os
import seaborn as sns
import warnings
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
from sklearn import metrics, preprocessing
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB, MultinomialNB
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.ensemble import VotingClassifier
from sklearn.naive_bayes import GaussianNB

Data preprocessing:

df = pd.read_csv('data_titanic.csv', header='infer')


# notice how in the age column there are missing values, we are going to treat these by categorizing into 30 or 10 based on the passenger name
n = df.shape[0]

Age = []                                                              
for i in range(n):
    if np.isnan(df.Age[i]): #si el campo está vacío
        if ('Mr' in df.Name[i]) or ('Mrs' in df.Name[i]) :
            Age.append(30)                                             # If Mr. or Mrs. in the name, then fill with 30.
        else:
            Age.append(10)                                             # Likely a child. So, fill with 10.
    else:
        Age.append(df.Age[i])
df.Age = pd.Series(Age)

#get rid of the columns which do not provide valuable info

df = df.drop(columns = ['PassengerId','Name','Ticket','Fare','Cabin'])

#Create a new column which will have the age data divided in 4 ranges

df['AgeCategory'] = pd.qcut(df.Age,4)  

# Convert into dummy variables and then remove the original variables.
df = pd.get_dummies(df.AgeCategory, drop_first=True,prefix='Age').join(df.drop(columns=['Age','AgeCategory']))
df = pd.get_dummies(df.Pclass, drop_first=True,prefix='Pclass').join(df.drop(columns=['Pclass']))
df = pd.get_dummies(df.SibSp, drop_first=True,prefix='SibSp').join(df.drop(columns=['SibSp']))
df = pd.get_dummies(df.Parch, drop_first=True,prefix='Parch').join(df.drop(columns=['Parch']))
df = pd.get_dummies(df.Sex, drop_first=True,prefix='Sex').join(df.drop(columns=['Sex']))
df = pd.get_dummies(df.Embarked, drop_first=True,prefix='Embarked').join(df.drop(columns=['Embarked']))

#set X variables and Y target variable

X = df.drop(columns=['Survived'])
Y = df.Survived

#Divide data into train and test
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=1234)

Hyperparameter optimization

I've chosen 5 classification models, now I'm going to do a Grid Search cross validation to get the best hyperparameters values for each model:

Decision Tree


depth_grid = np.arange(1,21)
min_samples_leaf_grid = np.arange(10,31)
max_leaf_nodes_grid = np.arange(2,21)
parameters = {'max_depth':depth_grid, 'min_samples_leaf':min_samples_leaf_grid, 'max_leaf_nodes':max_leaf_nodes_grid}

gridCV = GridSearchCV(DecisionTreeClassifier(), parameters, cv=10, n_jobs = -1)
gridCV.fit(X_train, Y_train);
best_depth = gridCV.best_params_['max_depth']
best_min_samples_leaf = gridCV.best_params_['min_samples_leaf']
best_max_leaf_nodes = gridCV.best_params_['max_leaf_nodes']

print("Tree best depth : " + str(best_depth))
print("Tree best min_samples_leaf : " + str(best_min_samples_leaf))
print("Tree best max_leaf_nodes : " + str(best_max_leaf_nodes))

DTC_best = DecisionTreeClassifier(max_depth=best_depth,min_samples_leaf=best_min_samples_leaf,max_leaf_nodes=best_max_leaf_nodes)
DTC_best.fit(X_train, Y_train);
Y_pred = DTC_best.predict(X_test)
print( "Tree best accuracy : " + str(np.round(metrics.accuracy_score(Y_test,Y_pred),3)))

KNN


n_neighbors_grid = np.arange(1, 21)
weights_grid = ['uniform', 'distance']
algorithm_grid = ['auto', 'ball_tree', 'kd_tree', 'brute']
parameters = {'n_neighbors': n_neighbors_grid, 'weights': weights_grid, 'algorithm': algorithm_grid}

gridCV = GridSearchCV(KNeighborsClassifier(), parameters, cv=10, n_jobs=-1)
gridCV.fit(X_train, Y_train)
best_n_neighbors = gridCV.best_params_['n_neighbors']
best_weights = gridCV.best_params_['weights']
best_algorithm = gridCV.best_params_['algorithm']

print("KNN best n_neighbors : " + str(best_n_neighbors))
print("KNN best weights : " + str(best_weights))
print("KNN best algorithm : " + str(best_algorithm))

KNN_best = KNeighborsClassifier(n_neighbors=best_n_neighbors, weights=best_weights, algorithm=best_algorithm)
KNN_best.fit(X_train, Y_train)

Y_pred_knn = KNN_best.predict(X_test)

accuracy = metrics.accuracy_score(Y_test, Y_pred_knn)
print("KNN best accuracy : " + str(np.round(accuracy, 3)))

Random Forest


n_estimators_grid = [50, 100, 150, 200]  
max_depth_grid = np.arange(1, 21)
min_samples_split_grid = [2, 5, 10]
min_samples_leaf_grid = [1, 2, 4]
max_features_grid = ['auto', 'sqrt', 'log2']

parameters = {
    'n_estimators': n_estimators_grid,
    'max_depth': max_depth_grid,
    'min_samples_split': min_samples_split_grid,
    'min_samples_leaf': min_samples_leaf_grid,
    'max_features': max_features_grid
}

gridCV = GridSearchCV(RandomForestClassifier(random_state=42), parameters, cv=10, n_jobs=-1)
gridCV.fit(X_train, Y_train)


best_n_estimators = gridCV.best_params_['n_estimators']
best_max_depth = gridCV.best_params_['max_depth']
best_min_samples_split = gridCV.best_params_['min_samples_split']
best_min_samples_leaf = gridCV.best_params_['min_samples_leaf']
best_max_features = gridCV.best_params_['max_features']

print("Random Forest best n_estimators: " + str(best_n_estimators))
print("Random Forest best max_depth: " + str(best_max_depth))
print("Random Forest best min_samples_split: " + str(best_min_samples_split))
print("Random Forest best min_samples_leaf: " + str(best_min_samples_leaf))
print("Random Forest best max_features: " + str(best_max_features))

RF_best = RandomForestClassifier(
    n_estimators=best_n_estimators,
    max_depth=best_max_depth,
    min_samples_split=best_min_samples_split,
    min_samples_leaf=best_min_samples_leaf,
    max_features=best_max_features,
    random_state=42
)
RF_best.fit(X_train, Y_train)

Y_pred_rf = RF_best.predict(X_test)
accuracy = metrics.accuracy_score(Y_test, Y_pred_rf)
print("Random Forest best accuracy: " + str(np.round(accuracy, 3)))

SVC


C_grid = [0.1, 1, 10, 100] 
kernel_grid = ['linear', 'poly', 'rbf', 'sigmoid']
degree_grid = [2, 3, 4, 5]
gamma_grid = ['scale', 'auto']

parameters = {
    'C': C_grid,
    'kernel': kernel_grid,
    'degree': degree_grid,
    'gamma': gamma_grid
}

gridCV = GridSearchCV(SVC(random_state=42), parameters, cv=10, n_jobs=-1)
gridCV.fit(X_train, Y_train)

best_C = gridCV.best_params_['C']
best_kernel = gridCV.best_params_['kernel']
best_degree = gridCV.best_params_['degree']
best_gamma = gridCV.best_params_['gamma']

print("SVM best C: " + str(best_C))
print("SVM best kernel: " + str(best_kernel))
print("SVM best degree: " + str(best_degree))
print("SVM best gamma: " + str(best_gamma))

SVM_best = SVC(
    C=best_C,
    kernel=best_kernel,
    degree=best_degree,
    gamma=best_gamma,
    random_state=42,
    probability = True
)
SVM_best.fit(X_train, Y_train)

Y_pred_svc = SVM_best.predict(X_test)

accuracy = metrics.accuracy_score(Y_test, Y_pred_svc)
print("SVM best accuracy: " + str(np.round(accuracy, 3)))

Gaussian NB


param_grid = {
      "var_smoothing": [1e-9, 1e-8, 1e-7, 1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1, 10, 100, 1000]
}

nb = GaussianNB()
grid_search_nb = GridSearchCV(nb, param_grid, cv=3, verbose=2, n_jobs=-1)
grid_search_nb.fit(X_train, Y_train)

y_pred_nb = grid_search_nb.predict(X_test)
accuracy_nb = accuracy_score(Y_test, y_pred_nb)
print(f"Accuracy NB: {accuracy_nb}")

Voting


Voting Soft:

voting_clf_soft = VotingClassifier(
   estimators=[
       ('Decission Tree', DTC_best),
       ('KNN', KNN_best),
       ('Random Forest', RF_best),
       ('SVM', SVM_best),
       ('GaussianNB', grid_search_nb)
   ],
   voting='soft'
)

voting_clf_soft.fit(X_train, Y_train)

y_pred_voting_soft = voting_clf_soft.predict(X_test)
accuracy_voting_soft = accuracy_score(Y_test, y_pred_voting_soft)
print(f"Accuracy: {accuracy_voting_soft}")

Voting Hard

voting_clf_hard = VotingClassifier(
   estimators=[
       ('Decission Tree', DTC_best),
       ('KNN', KNN_best),
       ('Random Forest', RF_best),
       ('SVM', SVM_best),
       ('GaussianNB', grid_search_nb)
   ],
   voting='hard'
)

voting_clf_hard.fit(X_train, Y_train)

y_pred_voting_hard = voting_clf_hard.predict(X_test)
accuracy_voting_hard = accuracy_score(Y_test, y_pred_voting_hard)
print(f"Accuracy: {accuracy_voting_hard}")



Conclusions 

Performance of individual models: 

Decision Tree 

A Decision Tree has been fitted using Grid Search with optimal parameters. 
The model's accuracy on the test set is 81.7%.

KNN 

A K-Nearest Neighbors (KNN) model has been fitted using Grid Search with optimal parameters. 

The model's accuracy on the test set is 81.0%.

Random Forest 

A Random Forest has been fitted using Grid Search with optimal parameters. 

The model's accuracy on the test set is 81.0%.

SVM 

A Support Vector Machine (SVM) has been fitted using Grid Search with optimal parameters. 

The model's accuracy on the test set is 81.3%.

Gaussian NB 

The model's accuracy on the test set is 77%.

Now we review the combination of these models through a Voting Classifier.

Soft Voting Accuracy: 82.1% This approach utilizes the prediction probabilities of each model and combines weighted predictions. It provides improved performance compared to the majority of individual models.

Hard Voting Accuracy: 81.7% This approach uses the majority prediction of individual models. Although slightly lower than the "soft" approach, it still represents an improvement compared to many individual models.

The use of a Voting Classifier has allowed leveraging the individual strengths of each model, thereby enhancing the overall performance of the classification system.

Comentarios

Publicar un comentario

Entradas populares de este blog

Convolutional Neural Network - Microplastic image classification PLASTISCAN

Imagen
  AI for Microplastic Images Classification   I want to present our final project for the Diploma in Advanced Artificial Intelligence offered by Samsung Innovation Campus and Universidad de Málaga . We decided to create our team under the name "Minerva Dev Team". We chose this name because we admire the figure of Minerva, goddess of wisdom, the arts, education, and justice. It seemed to us that this name best defined a team of five women, each so different yet contributing the best of themselves to create something beautiful from this experience. The project is called PlastiScan AI, a model designed, trained, and optimized with our own convolutional neural network. PlastiScan is responsible for analyzing images of water samples for the detection and classification of microplastic particles in them. Our purpose is to create a tool for research into such an important and current environmental issue, with the aim of helping to control pollution sources, understandin...
Read more »