Smartphone Sign Language Tutor

ISEF Category: Systems Software

Subcategory: Human/Machine Interface  ·  Difficulty: Intermediate  ·  Setup: Home Setup  ·  Time: 1 to 2 Months

The Hook

A phone can already count your steps and map your face. Why not check your fingerspelling, too? This project turns a smartphone into a coach for sign language practice. You can test whether adaptive feedback helps you learn faster than a plain video.

What Is It?

This project asks a simple question. Can a phone watch your hand, score your sign, and help you improve? The app uses hand-tracking software, like MediaPipe Hands, to find key points on your fingers and palm. Then a model compares your pose to a target sign and gives feedback.

Think of it like a mirror with a coach built in. A static video shows you what to do. An adaptive tutor reacts to your mistakes and changes the next task. That lets you study not just whether the app works, but whether it helps people learn better than a normal video lesson.

Why This Is a Good Topic

This is a strong science fair topic because you can test both the software and the learning effect. You can measure recognition accuracy, feedback speed, and user improvement over repeated practice. The project connects to accessibility, language learning, and human-computer interaction. You can build a meaningful prototype with a phone, open-source tools, and a clear comparison group.

Research Questions

• How does adaptive feedback affect fingerspelling accuracy compared with a static-video baseline?
• What is the effect of real-time hand-landmark scoring on learner improvement across practice sessions?
• Does adding error-specific feedback improve recognition of similar-looking signs more than generic feedback?
• To what extent does camera angle change the tutor’s accuracy in scoring hand shapes?
• Which confidence threshold gives the best balance between correct scoring and false feedback?
• What is the effect of hand-tracking latency on user performance during timed practice?

Basic Materials

• Smartphone with a front-facing camera.
• Tripod or phone stand.
• Free hand-tracking app or custom app built with MediaPipe Hands.
• Laptop or desktop computer for coding and analysis.
• Python installation.
• Webcam test area with plain background.
• Printed fingerspelling reference chart.
• Consent forms for any human participants.
• Notebook or spreadsheet for logging trials.
• Measuring tape or floor markers to keep camera distance consistent.

Advanced Materials

• Smartphone with a front-facing camera.
• Laptop with Python and model-training tools.
• MediaPipe Hands package.
• Open-source contrastive learning framework.
• Dataset of hand poses or recorded fingerspelling clips.
• External webcam for side-by-side validation.
• Calibration target for camera distance and angle checks.
• Statistical analysis software.
• Screen-recording tool for interface testing.
• Optional GPU access for faster model training.

Software & Tools

• MediaPipe Hands: Detects hand landmarks in real time for gesture tracking and scoring.
• Python: Handles data cleaning, model evaluation, and statistical analysis.
• Jupyter Notebook: Lets you inspect results, plot learning curves, and compare groups.
• OpenCV: Processes images and video frames for calibration and testing.
• ImageJ: Measures visual features if you need to compare hand position or frame quality.

Experiment Steps

1. Define the exact fingerspelling set you will test, and keep it small enough for reliable comparison.
2. Choose the one performance metric you will score first, such as landmark match, classification accuracy, or user learning gain.
3. Design a control condition, such as a static-video baseline, so you can compare the tutor against a simpler tool.
4. Plan how the app will give feedback, including what counts as a correct sign, a near miss, and a clear mistake.
5. Build a data collection plan that keeps camera angle, lighting, and hand position consistent across trials.
6. Prepare an analysis plan that compares improvement over time, not just first-attempt accuracy.

Common Pitfalls

• Training on your own hand only, which makes the tutor fail on different hand sizes, skin tones, or finger lengths.
• Letting camera angle drift between sessions, which changes landmark positions and confuses the model.
• Using too many similar signs at once, which makes the learning task harder to interpret and the accuracy scores noisier.
• Measuring only app accuracy and not learner improvement, which leaves out the educational part of the project.
• Giving vague feedback like “good” or “try again,” which makes it hard to tell whether adaptive coaching helps more than the baseline.

What Makes This Competitive

A strong version of this project goes beyond building an app. You compare learning outcomes with a clean control group, track more than one metric, and test whether feedback style changes performance on similar signs. Strong entries often report error rates, improvement curves, and confusion patterns, not just a single accuracy number. If you can show why the tutor helps or fails for certain sign types, your project gets much deeper.

Project Variations

• Test whether the tutor works better for ASL fingerspelling, number signs, or simple classroom vocabulary.
• Compare landmark-based scoring with a small image classifier to see which gives better feedback on device.
• Study whether left-hand and right-hand signing produce different accuracy or learning results in the same app.

Learn More

• MediaPipe documentation: Search the official MediaPipe site for hand-tracking examples and landmark output details.
• NIH PubMed: Search for review articles on sign language recognition, gesture tracking, and human-computer interaction.
• IEEE Access: Search for open-access papers on on-device gesture recognition and learning interfaces.
• MIT OpenCourseWare: Look for computer vision and machine learning lecture notes that explain feature extraction and model evaluation.
• arXiv: Search for recent preprints on hand pose estimation, contrastive learning, and sign language recognition.