CNN Code Documentation // ASL SOHA

High-level Overview

The CNN file is broken down into 3 distinct phases: data collection, training, and inference. The data collection phase is where sample signs for each letter were collected by placing our fingers in front of the webcam. In the training phase, the model used these samples to detect patterns and learn how to map each ASL hand signal to a corresponding English letter. After learning these patterns, the model predicted which English letter was signaled by a sign in real time during the inference stage.

`collect_data()`

The collect_data() method was programmed to open a new camera window within the user's laptop and continuously search for a hand. When the hand was detected, 21 different landmarks (positions) on the hand were searched for. In order to standardize the positions of the landmarks across different camera distances and angles so the model receives consistent data for each letter, I decided to calculate the positions by their displacement from the wrist. The 3 displacements were the x-displacement, y-displacement, and z-displacement in 3D space.

X, Y, Z coordinate displacement from wrist to normalize across camera distances and angles:

Diagram of a human hand with the 21 MediaPipe landmarks numbered 0 to 20, and a table naming each one — // Hand landmarks

Python imports for the mediapipe library, its hands solution, and its drawing utilities

The mediapipe library was used to detect the hand in the video frame, and extract the exact landmark coordinates.

Code that opens the webcam with cv2.VideoCapture and initializes the dataset dictionary with an empty list for each label

The first line cap = cv2.VideoCapture(0) initializes the video window to read frames continuously. The dataset is initialized to an empty dictionary that maps each label (English character) to a list of frames; each frame consists of the 21 different landmarks, represented by a list of 3 positions.

Code that flips and converts each frame, detects hand landmarks, draws them, and subtracts the wrist position from every landmark

Note: if results.multi_hand_landmarks checks if a hand is detected. The hand_landmark.landmark list contains 21 landmarks, each with an (x, y, z) position. The wrist is the first landmark accessed.

Subsequently, the user is prompted to enter the letter in the video window that corresponds to the sign they were holding up, so that the model can learn how to predict the English letter from a visual sign. After collecting each frame and its corresponding letter, the dictionary was parsed into JSON and loaded into a datasheet file.

Code that waits for a key press, checks for a letter A to Z, and stores the landmarks under that letter

The window waits for a key to be pressed with the waitKey() method. The program then checks for a valid letter (A-Z) to be entered. If a hand is detected, the landmarks of the frame are added to the dataset for that particular letter. The data collection is exited when 'Q' is entered. Finally, the final dataset with all the landmarks for each letter is loaded as a JSON into the data file.

Code that writes the finished dataset of landmarks to a JSON file

One Adjustment I Made

Code that loads the existing dataset if it exists, otherwise creates empty lists for each letter

The new code was added to prevent losing old data when training the model on new letters. Because I trained the models on certain letters at first, and then the rest of the letters the second time, simply resetting the dataset would erase the old data, leading to the model losing training on the previous letters.

The code checks if the dataset already exists. If so, it extracts it into the dictionary and initializes the remaining letters. If not, it creates a new empty dictionary with empty lists for each letter.

`train_model()`

The train_model() method uses the dataset created by the previous method, initializes the CNN model, and trains it on the collected data following the user's exit from data collection. When training the model, the inputs are the landmarks, and the outputs are the English letters represented by those landmarks on the sign in the webcam.

Code that loads the dataset, builds a label map, and fills the input and output lists

The code above loads the dataset from the file back into a dictionary. A label map is created that maps each label (letter) to its index in the CLASSES list. This is so that its index can be accessed when loading it into the output list for training.

We then looped through each of the frames of landmarks in the dataset, adding them to the input (X) list. We added its corresponding label (index of letter) to the corresponding slot in the output (y) list. We do this because the model can only take in numerical values, so it will predict based on integer labels; specifically, it gives the probability for that letter in the index equal to that label.

Code that splits the inputs and outputs into training and testing sets

The piece of code above helps to split the input and output data we gathered into training and testing data. The majority of it is used to train the model, but the remaining inputs and outputs are used to verify its accuracy and validity.

Model Architecture

Keras code defining the CNN as a Sequential model with convolution, batch normalization, dropout, flatten, and dense layers

The code above initializes the CNN. The Sequential method was used to establish the different layers of the vision model.

2D convolution layer with 32 3x3 filters (layer 1) slides across the input frames and detects low-level patterns. 'Relu' only keeps positive values in the input to prevent linearity.
BatchNormalization() (layer 2) helps to scale the input and make training more efficient.
Dropout(.2) (layer 3) was used to turn off 20% of neurons to limit the model's ability to memorize and prevent overfitting.
Another 2D convolution layer with 64 3x1 filters (layer 4), best used to catch vertical patterns.
Flatten (layer 5) is used to convert the feature maps to a long, 1D list.
Dense layer (layer 6) contains 64 neurons and receives the flattened feature map to extract deeper features.
Another dropout layer (layer 7).
The final dense layer (layer 8) outputs scores for each of the letters and contains a softmax to convert those scores to probabilities (confidence).

CNN Architecture Visualization

Block diagram of the CNN: convolution, batch norm, dropout, a second convolution, flatten, dense, dropout, and softmax output probabilities for each letter — // CNN architecture diagram

The model.fit() trains the model on our collected input and output data; we decided to train it in batches of 8 frames at a time with 30 loops.

`run_inference()`

The run_inference() method takes the newly trained CNN and has it make predictions by converting human ASL signs in the webcam into English letters in real time. The first step in inference is to prepare the input data for the model to predict based on. This requires extracting the landmarks for that to pass into the model.

Code that detects the hand, extracts landmarks relative to the wrist, and reshapes them with NumPy

The code above analyzes the hand, extracts the landmarks, and loads the x, y, and z displacements from the wrist into the input array, which is then reshaped using NumPy.

Code that passes the landmarks to the model and uses argmax to pick the most likely letter

In the code above, that input list of landmarks is then passed into the model for a prediction. The model will return a list of probabilities for each label (letter index). To find the label with the highest probability to get the model's letter prediction, we use argmax, which returns the index of the highest value in the tensor.

After extracting the highest-confidence prediction, we use that label to get the correct letter, assuming the model was confident enough.

Observations on CNN Results

CNN Results Doc

After training the model on the data I collected, which initially consisted of 100 frames (sets of 21 landmarks) for each letter, the accuracy reached was around 97.9%. Therefore, I decided to collect more frames for each letter because I thought that would help the model learn more deeply. However, after collecting 100 more frames for each letter, the accuracy stayed roughly the same (even slightly lower).

To find the issue, I tested the model in live inference and noticed that the model was confusing multiple letters with each other. It was confusing (M, N), (U, V), (Y, I), (A, I), (T, M). This is because of the very similar landmark positions of these letters despite being entirely different signs. Therefore, I collected 50 more frames for each of the letters above, trying to exaggerate and clearly show the model the differences between the signs. Surprisingly, I noticed that the model's accuracy dropped slightly. I predicted that the model was only starting to learn the differences between the letters as the 30th epoch was up, so I added 20 more epochs (to make it 50).

This addition allowed the model's accuracy to bump up to roughly 99%. I avoided adding more epochs to prevent overfitting.

Post-Training Addition

One issue I noticed was that the model was continuously outputting new predictions on every frame, meaning if the user changed their finger positions slightly, the model would immediately switch to a different letter.

To ensure the model outputs a prediction only when the user's hand is stable, I tracked a candidate streak. This streak updates if the current prediction matches the previous one, and only becomes the model's final prediction if the streak reaches a value of 3. Therefore, the model only outputs a prediction if it guesses that exact letter 3 times in a row. (Reference the code snippet below)

Code that accepts a prediction only above 0.75 confidence and tracks a candidate streak, increasing it when the prediction matches the previous one and resetting it to 1 when it changes