Convolutional Neural Network - Microplastic image classification PLASTISCAN

 

AI for Microplastic Images Classification

 

I want to present our final project for the Diploma in Advanced Artificial Intelligence offered by Samsung Innovation Campus and Universidad de Málaga.

We decided to create our team under the name "Minerva Dev Team". We chose this name because we admire the figure of Minerva, goddess of wisdom, the arts, education, and justice. It seemed to us that this name best defined a team of five women, each so different yet contributing the best of themselves to create something beautiful from this experience.

The project is called PlastiScan AI, a model designed, trained, and optimized with our own convolutional neural network. PlastiScan is responsible for analyzing images of water samples for the detection and classification of microplastic particles in them. Our purpose is to create a tool for research into such an important and current environmental issue, with the aim of helping to control pollution sources, understanding how and to what extent it affects, or understanding its degradation.

Creating the model has not been easy; handling such a large dataset with our own resources has been a truly tough task that has resulted in sleepless nights and leaving the equipment running.

In addition to this, we add the challenge of creating a web application from scratch, design, structure, server configuration... where we can fully integrate it, not only to provide more information about the project and the team but also to offer a real test of its functionality and to try it out by uploading your own photo. Here is the link, we encourage you to try it and give us your opinion:

https://www.plastiscan.org

None of us had previous experience in this regard, so it has been a tough journey, but after many sleepless nights, we did it. We are very proud that in such a short time, we have not only completed the project of creating the model but also been able to showcase our work in a practical way for anyone interested.

Of course, although it is a first prototype with many improvements to be made both in design and in the model itself, we leave you the GitHub repository.

https://lnkd.in/eXaYJygu

We make our project available to researchers, organizations, and anyone or entity who wants to advance in the same direction, sharing it under a Creative Commons license (CC BY-NC-SA 4.0).

Minerva Dev Team:

Jessica Piñas
Cristina Morales LL
Laura Moreno Fernández
Adriana Porrero Diaz
Verónica Martínez

Now, lets dive into the fun part!

We came across a dataset that initially consisted of over 3000 images of microplastics classified by shape, color, component, and diameter. We found the data on this Japanese website:

https://corp.pirika.org/en/public-survey-results/

 

The code below broadly represents the initial exploratory data analysis, implementation of image loading functions, normalization, and resizing to feed into the CNN model, handling of columns with unbalanced data, a as well as the model construction, compilation and training.

 

1. EDA

Summary of the main steps taken for data preprocessing and cleaning

columns_to_keep = ['Image', 'diameter / mm', 'Color', 'Shape', 'Component']

df_filtered = df[columns_to_keep]

df_filtered.head()

#Below function loads the image, resizes it and converts it into a numpy array:

def load_and_resize_image(image_path, target_size=(299, 299)):

    try:

        with Image.open(image_path) as img:

            img_resized = img.resize(target_size)

            img_array = np.array(img_resized)

            img_array_normalized = img_array.astype('float32') / 255.0

            return img_array_normalized

    except IOError:

        print(f"Image could not be loaded: {image_path}")

        return None

# Función create dataset

def create_dataset(data_df, images_folder):

    dataset = []

    # Eliminar NaN en 'Image'

    data_df = data_df.dropna(subset=['Image'])

    for index, row in data_df.iterrows():

        image_filename = str(row['Image'])  # str

        image_path = os.path.join(images_folder, image_filename)

        image = load_and_resize_image(image_path, target_size=(299, 299))

        if image is not None:

            image_data = (image,

                          row['diameter / mm'],

                          row['Color'],

                          row['Shape'],

                          row['Component'])

            dataset.append(image_data)

    return dataset

 

#new dataset with the processed images:

df_with_images = create_dataset(df_filtered, images_folder)

df_nuevo = pd.DataFrame(df_with_images, columns=['Columna1', 'Columna2', 'Columna3', 'Columna4', 'Columna5'])

df_nuevo.columns = ['Image_normalized', 'diameter / mm', "Color","Shape", "Component"]

 

 

 

df_nuevo['diameter / mm'].replace('-', np.nan, inplace=True)

df_nuevo['diameter / mm'] = df_nuevo['diameter / mm'].str.replace(',', '.')

df_nuevo['diameter / mm'] = df_nuevo['diameter / mm'].astype(float)

 

df_nuevo = df_nuevo.dropna(subset=["Shape", "Component"])

df_nuevo = df_nuevo.dropna(subset=["diameter / mm"])

 

rows_to_drop = df_nuevo[df_nuevo['Shape'].str.contains("-")].index

df_nuevo = df_nuevo.drop(rows_to_drop)

rows_to_drop = df_nuevo[df_nuevo['Component'].str.contains("-")].index

df_nuevo = df_nuevo.drop(rows_to_drop)

 

 #removing unbalance data from column Shape

df_nuevo = df_nuevo[~df_nuevo['Shape'].isin(["Pellet", "Sphere"])]

 

# Removing PU, PVC, EVA, PDMS, Polyacetal, ABS, Polychloroprene, PVA y PMMA from column Component

suma_por_componente = df_nuevo['Component'].value_counts()

componentes_filtrados = suma_por_componente[suma_por_componente >= 25].index

df_nuevo = df_nuevo[df_nuevo['Component'].isin(componentes_filtrados)]

 

 

# transforming diameter

df_nuevo['Diameter / µm'] = df_nuevo['diameter / mm'] * 1000

df_nuevo.drop(columns=['diameter / mm'], inplace=True)

df_nuevo.info()

 

 

# Adding the category column based on the diameter

def asignar_categoria(diametro):

    if diametro >= 0.001 and diametro <= 1:

        return 'Nanoplastic'

    elif diametro > 1 and diametro <= 1000:

        return 'Microplastic'

    elif diametro > 1000 and diametro <= 10000:

        return 'Mesoplastic'

    elif diametro > 10000:

        return 'Macroplastic'

 

 

df_nuevo['Category'] = df_nuevo['Diameter / µm'].apply(asignar_categoria)

 

 

 

2. Creating the CNN 

 

 

import tensorflow as tf

from tensorflow.keras.utils import to_categorical

from sklearn.model_selection import train_test_split, GridSearchCV

from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense, Dropout

from tensorflow.keras.models import Model

from tensorflow.keras.optimizers import Adam

from tensorflow.keras.callbacks import EarlyStopping, TensorBoard, ModelCheckpoint

import numpy as np

import json

import datetime

import warnings

warnings.filterwarnings('ignore')

%reload_ext tensorboard

 

 

 

 

# Applying one-hot encoding for categorical columns

shape_one_hot = pd.get_dummies(df_nuevo['Shape'], prefix='Shape')

color_one_hot = pd.get_dummies(df_nuevo['Color'], prefix='Color')

component_one_hot = pd.get_dummies(df_nuevo['Component'], prefix='Component')

category_one_hot = pd.get_dummies(df_nuevo['Category'], prefix='Category')

df_final_one_hot = df_nuevo.join([shape_one_hot, color_one_hot, component_one_hot, category_one_hot])

 

 

df_final_one_hot.to_csv('dataset_procesado_V2.csv', index=False)

 

X = np.stack(df_final_one_hot['Image_normalized'].values)

y = np.hstack([shape_one_hot.values, color_one_hot.values, component_one_hot.values, category_one_hot.values])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

 

X = np.stack(df_final_one_hot['Image_normalized'].values)

y = np.hstack([shape_one_hot.values, color_one_hot.values, component_one_hot.values, category_one_hot.values])

X_train, X_test, y_shape_train, y_shape_test = train_test_split(X, shape_one_hot, test_size=0.2, random_state=42)

_, _, y_color_train, y_color_test = train_test_split(X, color_one_hot, test_size=0.2, random_state=42)

_, _, y_component_train, y_component_test = train_test_split(X, component_one_hot, test_size=0.2, random_state=42)

_, _, y_category_train, y_category_test = train_test_split(X, category_one_hot, test_size=0.2, random_state=42)

param_grid = {

    'batch_size': [32, 64],

    'epochs': [5 ,10],

    'learning_rate': [0.001, 0.01]

}

# Defining the callbacks

callbacks = [

    EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True),

     TensorBoard(log_dir='C:/Users/laura/Python Laura/SEGUNDA PARTE DEL CURSO/PROYECTO FINAL/PROTOTIPOS/V 1.0/logscnn-hip3', histogram_freq=1),

    ModelCheckpoint(filepath='best_modelV1.5BUENO.h5', monitor='val_loss', save_best_only=True)

]

def create_compile_model(learning_rate=0.001):

    input_shape = (299, 299, 3)

    inputs = Input(shape=input_shape)

    conv1 = Conv2D(32, (3, 3), activation='relu')(inputs)

    pool1 = MaxPooling2D((2, 2))(conv1)

    conv2 = Conv2D(64, (3, 3), activation='relu')(pool1)

    pool2 = MaxPooling2D((2, 2))(conv2)

    conv3 = Conv2D(128, (3, 3), activation='relu')(pool2)

    pool3 = MaxPooling2D((2, 2))(conv3)

    flatten = Flatten()(pool3)

   

    # Adding dense layers for each category (and dropout for overfitting)

    shape_dense = Dense(64, activation='relu')(flatten)

    shape_dropout = Dropout(0.5)(shape_dense)

    shape_output = Dense(3, activation='softmax', name='shape_output')(shape_dropout)

   

    color_dense = Dense(64, activation='relu')(flatten)

    color_dropout = Dropout(0.5)(color_dense)

    color_output = Dense(9, activation='softmax', name='color_output')(color_dropout)

   

    component_dense = Dense(64, activation='relu')(flatten)

    component_dropout = Dropout(0.5)(component_dense)

    component_output = Dense(7, activation='softmax', name='component_output')(component_dropout)

   

    category_dense = Dense(64, activation='relu')(flatten)

    category_dropout = Dropout(0.5)(category_dense)

    category_output = Dense(3, activation='softmax', name='category_output')(category_dropout)

   

    model = Model(inputs=inputs, outputs=[shape_output, color_output, component_output, category_output])

    optimizer = Adam(learning_rate=learning_rate)

    model.compile(optimizer=optimizer,

                  loss={'shape_output': 'categorical_crossentropy',

                        'color_output': 'categorical_crossentropy',

                        'component_output': 'categorical_crossentropy',

                        'category_output': 'categorical_crossentropy'},

                  metrics=['accuracy'])

    return model

# Iterate over hiperparámetros

results = []

for batch_size in param_grid['batch_size']:

    for epochs in param_grid['epochs']:

        for learning_rate in param_grid['learning_rate']:

            print(f"Training model with batch_size={batch_size}, epochs={epochs}, learning_rate={learning_rate}")

            model = create_compile_model(learning_rate)

            history = model.fit(X_train,

                                [y_shape_train, y_color_train, y_component_train, y_category_train],

                                batch_size=batch_size,

                                epochs=epochs,

                                callbacks=callbacks,

                                validation_split=0.2)

           

            # Saving the results

            val_loss = np.mean(history.history['val_loss'][-5:])

            val_accuracy = []

            for output_name in model.output_names:

                val_accuracy_output = np.mean(history.history[f'val_{output_name}_accuracy'][-5:])

                val_accuracy.append(val_accuracy_output)

            results.append((val_loss, val_accuracy, batch_size, epochs, learning_rate))

# Finding the best hiperparameter config

best_result = sorted(results, key=lambda x: x[0])[0]  # Ordenar por val_loss

print(f"Mejores hiperparámetros: batch_size={best_result[2]}, epochs={best_result[3]}, learning_rate={best_result[4]} con val_loss={best_result[0]}")

for i, output_name in enumerate(['shape', 'color', 'component', 'category']):

    print(f"Accuracy {output_name}: {best_result[1][i]}")

 

 

 

best_model = create_compile_model(best_result[4])  # Creamos el mejor modelo con los mejores hiperparámetros

best_model.fit(X_train,

               [y_shape_train, y_color_train, y_component_train, y_category_train],

               batch_size=best_result[2],

               epochs=best_result[3],

               callbacks=callbacks,

               validation_split=0.2)

 

 

 

def plot_history(history):

    fig, ax = plt.subplots(5, 2, figsize=(20, 25))

    outputs = ['shape_output', 'color_output', 'component_output', 'category_output']

    for i, output in enumerate(outputs):

        ax[i, 0].plot(history.history[f'{output}_loss'], label='train')

        ax[i, 0].plot(history.history[f'val_{output}_loss'], label='val')

        ax[i, 0].set_title(f'{output} Loss')

        ax[i, 0].set_xlabel('Epochs')

        ax[i, 0].set_ylabel('Loss')

        ax[i, 0].legend()

        ax[i, 1].plot(history.history[f'{output}_accuracy'], label='train')  # Asegúrate de que 'accuracy' es correcto

        ax[i, 1].plot(history.history[f'val_{output}_accuracy'], label='val')  # Ajusta este nombre si es necesario

        ax[i, 1].set_title(f'{output} Accuracy')

        ax[i, 1].set_xlabel('Epochs')

        ax[i, 1].set_ylabel('Accuracy')

        ax[i, 1].legend()

    # Graficamos la pérdida total

    ax[4, 0].plot(history.history['loss'], label='train')

    ax[4, 0].plot(history.history['val_loss'], label='val')

    ax[4, 0].set_title('Total Loss')

    ax[4, 0].set_xlabel('Epochs')

    ax[4, 0].set_ylabel('Loss')

    ax[4, 0].legend()

    plt.tight_layout()

    plt.show()

plot_history(history)