Supervised Learning | Binary Classification with Random Forest Classifier

Binary Classification with Random Forest Classifier

We will walk through the process of deploying a Random Forest Classifier model.

We'll use a model trained on the Titanic dataset to predict passenger survival and explain each step along the way.

Step 1: Model Selection and Training

A Random Forest Classifier was chosen as the model for this binary classification problem. The model was trained on the preprocessed dataset with a random seed for reproducibility.

The goal was to predict whether a passenger survived or not based on various features.

Step 2: Data Preprocessing

Before training the model, I preprocessed the data to clean and prepare it for analysis. This included:

• Normalizing passenger names.
• Extracting ticket numbers and ticket items.
• Handling missing data in age and embarked columns.
• One-hot encoding categorical variables like "Sex" and "Embarked.

Step 3: Feature Selection

The model used a combination of numerical features like "Pclass," "Age," "SibSp," "Parch," and "Fare," along with one-hot encoded categorical features like "Sex" and "Embarked." These features were selected as the input variables to predict the "Survived" target variable.

Step 4: Model Evaluation

The model's performance was evaluated using metrics such as accuracy, precision, recall, and F1-score. The overall accuracy achieved on the test dataset was approximately 82.12%. The classification report provides additional details on precision and recall for both the "Survived" and "Not Survived" classes.

Step 5: Feature Importance

The model was analyzed to understand which features had the most significant impact on predictions. The top three important features, in descending order of importance, were:

• Sex_male with an importance of approximately 27.33%
• Fare with an importance of approximately 27.21%
• Age with an importance of approximately 25.27%

These insights can help us understand which factors were crucial in determining whether a passenger survived or not.

Step 6: Binning and Correlation Analysis

The "Pclass" feature was binned into categories, and a point-biserial correlation coefficient was calculated between "Survived" and "Cabin." This analysis provides additional information about the relationships between variables in the dataset.

Step 7: Conclussion

The Random Forest Classifier model appears to be a reasonably good predictor for the Titanic survival prediction task, achieving an accuracy of approximately 82.12% on the test dataset. The most influential factors for predicting survival are gender (Sex_male), fare (Fare), and age (Age). It's important to note that further analysis and fine-tuning of the model could potentially lead to improved performance

----------------------------------------------------------------------------------------------------------------------------

# Import necessary libraries

import pandas as pd
import matplotlib.pylab as plt
import numpy as np
from scipy import stats

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report

import seaborn as sns
%matplotlib inline 

# Read the training and test datasets

https://www.kaggle.com/competitions/titanic/data

# Clean and modify the data

def preprocess(df):

    """ Clean and modify the data """

    df = df.copy()
    
    def normalize_name(x):
        return " ".join([v.strip(",()[].\"'") for v in x.split(" ")])
    
    def ticket_number(x):
        return x.split(" ")[-1]
        
    def ticket_item(x):
        items = x.split(" ")
        if len(items) == 1:
            return "NONE"
        return "_".join(items[0:-1])
    
    df["Name"] = df["Name"].apply(normalize_name)
    df["Ticket_number"] = df["Ticket"].apply(ticket_number)
    df["Ticket_item"] = df["Ticket"].apply(ticket_item)                     
    return df
    
preprocessed_train_df = preprocess(train_df)
preprocessed_test_df = preprocess(test_df)

# Handle missing data in the "Age" and "Embarked" columns

train_df['Age'].fillna(train_df['Age'].median(), inplace=True)
train_df['Embarked'].fillna(train_df['Embarked'].mode()[0], inplace=True)

# Encode categorical variables ("Sex" and "Embarked") using One-Hot Encoding

encoder = OneHotEncoder(drop='first', sparse=False)
encoded_features = encoder.fit_transform(train_df[['Sex', 'Embarked']])

feature_names = encoder.get_feature_names_out(input_features=['Sex', 'Embarked'])

train_df_encoded = pd.concat([train_df, pd.DataFrame(encoded_features, columns=feature_names)], axis=1)

# Select features and target variable for modeling
features = ['Pclass', 'Age', 'SibSp', 'Parch', 'Fare'] + list(feature_names)
X = train_df_encoded[features]
y = train_df_encoded['Survived']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)

#Model Evaluation

y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
report = classification_report(y_test, y_pred)

feature_importances = pd.DataFrame({'Feature': features, 'Importance': model.feature_importances_})

feature_importances = feature_importances.sort_values(by='Importance', ascending=False)

binary_target = train_df["Survived"]
continuous_v = train_df["Cabin"]
r_pb, _ = stats.pointbiserialr(binary_target, continuous_v)
print(f"Point-Biserial Correlation Coefficient: {r_pb}")

Now you can predict the survival of the test data:

preprocessed_test_df = preprocess(test_df)

encoded_features_test = encoder.transform(preprocessed_test_df[['Sex', 'Embarked']])

feature_names_test = encoder.get_feature_names_out(input_features=['Sex', 'Embarked'])

test_df_encoded = pd.concat([preprocessed_test_df, pd.DataFrame(encoded_features_test, columns=feature_names_test)], axis=1)

test_features = ['Pclass', 'Age', 'SibSp', 'Parch', 'Fare'] + list(feature_names_test)


test_df_encoded['Age'].fillna(train_df['Age'].median(), inplace=True)
test_df_encoded['Fare'].fillna(train_df['Fare'].median(), inplace=True)  # You might need to replace this with the median from the training data

# Use the trained model to make predictions on the test data
test_predictions = model.predict(test_df_encoded[test_features])

# The variable test_predictions now contains the predicted survival outcomes (0 or 1) for passengers in the test dataset.


# you can save the updated test_df with predictions to a CSV file if needed
test_df.to_csv('test_predictions.csv', index=False)