Phase 3 // ASL SOHA

In-depth ASL SOHA CNN Code Documentation

All code can be found on our GitHub repository

Phase 3 was where we started getting both the hardware and software sides of SOHA to a point where they could actually be connected together.

On the hardware side, we finished sewing and assembling the robotic hand, making sure the tendons were routed through the fingers correctly so that the servos could control each joint. We also connected the power supply to the servo system, which gave us a more stable setup for testing the hand instead of powering everything separately.

The finished sewn robotic hand, tendons routed through the fingers — // Finished sewn robotic hand

The power supply, driver board, and servos wired up on the workbench — // Power supply and servo wiring setup

At the same time, we worked more on the computer vision side by building the CNN that recognizes ASL signs through a webcam. The code is divided into three main sections: data collection, model training, and real-time inference.

For the data collection section, the webcam is opened using OpenCV, and MediaPipe searches each frame for a hand. Once a hand is detected, the program finds 21 landmarks across the wrist and fingers.

Code showing the webcam being opened with OpenCV and MediaPipe detecting a hand in each frame — // Webcam input and MediaPipe hand detection

Instead of using the raw landmark positions, we normalized every point relative to the wrist. The wrist is landmark 0, so the program subtracts its x, y, and z coordinates from every other landmark. This made the data more consistent because the same sign would not completely change just because someone moved their hand to a different part of the camera. Measuring from the wrist also helps standardize the landmarks across different camera distances and angles, so the model receives more consistent data for each letter.

Code that subtracts the wrist position from every hand landmark — // Wrist normalization

To build the dataset, we hold up a sign and press its corresponding letter on the keyboard. The program then stores the 21 landmarks under that letter and eventually saves everything into asl_dataset.json. Right now, the model is trained on all 26 letters.

The data collection window on a laptop, with the hand landmark skeleton drawn over a raised hand — // Webcam showing the landmark skeleton and sample counters

Diagram of a human hand with the 21 MediaPipe landmarks numbered 0 to 20, and a table naming each one — // Hand landmarks

After collecting the data, the training section converts every letter into a numerical label and reshapes each sample into a 21 × 3 × 1 input. The data is then split so that 80% is used for training and 20% is used for validation.

The CNN itself uses two convolutional layers, followed by dropout, flattening, and dense layers.

Block diagram of the CNN: convolution, batch norm, dropout, a second convolution, flatten, dense, dropout, and softmax output probabilities for each letter — // CNN architecture diagram

Keras code defining the CNN layers, then compiling and training the model — // The CNN architecture code

We trained the model for 30 epochs with a batch size of 8. After training, the program moves into the real-time inference section.

During inference, the webcam again detects the hand, calculates the same wrist-relative landmarks, and reshapes them before sending them into the model. The program then uses np.argmax() to find which letter has the highest probability. We also added a 75% confidence threshold, so if the model is not confident enough, it displays "Uncertain..." instead of forcing a prediction.

Code that picks the most likely letter and shows Uncertain when confidence is below 75 percent — // Confidence percentage

By the end of Phase 3, we had both sides of SOHA mostly working on their own. The robotic hand was fully sewn, the tendons and servos were connected, the power supply was set up, and the CNN could recognize ASL signs in real time.

The main thing left now is combining everything.

In Phase 4, we will focus on the Raspberry Pi code that connects the two systems. Right now, the CNN can output a letter such as "B" or "Y", while the hand can be controlled through its servos. The Raspberry Pi will act as the bridge between them. The goal is for the final system to work like this:

Webcam → MediaPipe landmarks → CNN prediction → Raspberry Pi → servo commands → robotic hand

Once that connection is finished, the software will no longer just recognize a letter on a screen. The prediction will actually be able to trigger the correct movement on the physical hand, which will bring the full SOHA system much closer to working as one device.