Webcam Pose Estimation With Test-Time Adaptation

ISEF Category: Robotics and Intelligent Machines

Subcategory: Machine Learning  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

A pose model can look smart in perfect lab lighting, then fail the moment a shadow crosses the arm. That makes this topic powerful for a science fair. You are not just training a model, you are testing how well it keeps working when reality gets messy. That is the kind of problem robots face outside the demo room.

What Is It?

This project asks a simple question with a hard answer, can a vision model keep tracking arm keypoints when the camera view gets worse? Keypoints are points on the body, like elbow, wrist, or shoulder. The model tries to predict where those points are from one webcam image. Your job is to see whether a model that keeps learning during test time does better than a static model that never changes after training.

Think of it like taking a quiz with open notes after class. A static model walks in with one fixed memory. A test-time-training model gets a chance to adjust itself from the new scene using masked reconstruction loss, which means it learns by filling in missing parts of the image or features. You can compare the two models under normal lighting, harsh shadows, partial occlusion, and background clutter, then measure which one makes smaller keypoint errors.

Why This Is a Good Topic

This is a strong science fair topic because you can test one clear idea, adaptation should help when the camera view changes. You can build measurable trials, compare a baseline against an adaptive model, and use real metrics like keypoint mean squared error. The topic connects to real robot vision, teleoperation, and motion tracking. You can learn model evaluation, experimental control, and error analysis without needing a giant lab setup.

Research Questions

• How does test-time training change keypoint mean squared error under harsh lighting compared with a static baseline?
• What is the effect of partial arm occlusion on pose accuracy for adaptive and non-adaptive models?
• Does masking more image regions during test-time adaptation improve or hurt keypoint prediction on new lighting conditions?
• To what extent does background clutter change the size of the adaptation gain over a static model?
• Which keypoints, elbow, wrist, or shoulder, benefit most from test-time adaptation?
• How does the number of adaptation steps affect error before the model starts overfitting to one scene?

Basic Materials

• One webcam with fixed resolution settings.
• One computer that can run Python.
• A dataset of arm images or videos with labeled keypoints.
• Python packages for computer vision and machine learning.
• A way to create lighting changes, such as desk lamps, blinds, or cardboard shadows.
• A plain wall, poster board, or cloth backdrop.
• A notebook for logging trial conditions and errors.

Advanced Materials

• One university lab computer or workstation with a GPU.
• One webcam or RGB camera with manual exposure control.
• A pose dataset with arm keypoints and domain shifts.
• A deep learning framework such as PyTorch.
• Image annotation software for checking keypoint labels.
• A controlled lighting rig or adjustable LED panels.
• Occlusion objects, such as gloves, sleeves, cardboard cutouts, or meshes.
• A tripod or camera mount for repeatable framing.

Software & Tools

• Python: Runs the training, inference, and evaluation code for your pose model.
• PyTorch: Lets you build the baseline model and the test-time training version.
• OpenCV: Helps you process webcam frames, add masks, and measure image changes.
• ImageJ: Lets you inspect frames and compare visual differences in lighting or occlusion.
• Jupyter Notebook: Helps you organize experiments, plots, and error tables in one place.

Experiment Steps

1. Define the exact pose problem, the arm keypoints you will predict, and the baseline model you will compare against.
2. Choose one test-time adaptation rule, then decide what image or feature information the model will use to update itself at deployment.
3. Plan your lighting and occlusion conditions so each trial changes only one factor at a time.
4. Build a fair evaluation set, then lock it before you start tuning so you do not accidentally train on your test scenes.
5. Decide how you will score success, such as keypoint MSE, per-joint error, or error change relative to the baseline.
6. Map out the analysis you will use to check whether the adaptive model improves consistently or only in a few scenes.

Common Pitfalls

• Testing the adaptive model on scenes that also appeared in training, which makes the gain look larger than it really is.
• Changing camera position between trials, which mixes viewpoint error with lighting or occlusion effects.
• Comparing models on different frame sets, which breaks the fairness of the baseline test.
• Letting the model adapt too long on one scene, which can cause overfitting to a single lighting pattern.
• Measuring only one summary score, which can hide the fact that some keypoints improve while others get worse.

What Makes This Competitive

A competitive version goes beyond a simple before-and-after comparison. You would test several kinds of scene shift, then show where adaptation helps, where it fails, and why. Strong analysis would include per-keypoint error, confidence calibration, and statistics across repeated trials. If you can explain the tradeoff between fast adaptation and overfitting, your project starts to look like real research.

Project Variations

• Test the same idea on hand keypoints instead of arm keypoints to see whether smaller joints react differently to occlusion.
• Compare masked-reconstruction adaptation with another test-time update rule, such as entropy minimization, to see which handles lighting shifts better.
• Use synthetic shadows or cutout occluders to create a controlled stress test and measure how accuracy drops as the scene gets harder.

Learn More

• PyTorch Tutorials: Free guides for building and evaluating neural networks, found by searching the official PyTorch tutorials site.
• OpenCV Documentation: Free computer vision reference for webcam capture, image masking, and frame processing, found on the official OpenCV docs.
• COCO Keypoints Dataset: Widely used human pose data and annotations, found by searching the COCO dataset page.
• Papers With Code: Search for human pose estimation and test-time adaptation papers to compare methods and metrics.
• PubMed: Search review articles on vision-based motion tracking and human pose estimation for background reading.
• MIT OpenCourseWare: Search computer vision and machine learning course materials for free lecture notes and assignments.