Webcam Arousal Tracking with Facial Micro-Expressions

ISEF Category: Behavioral and Social Sciences

Subcategory: Behavioral Neuroscience  ·  Difficulty: Advanced  ·  Setup: School Lab  ·  Time: 1 to 2 Months

The Hook

Your face can change before you name the feeling. Tiny expression shifts can act like a dashboard needle for mental load. In this project, you use a webcam, MediaPipe, and a small CNN to see whether those shifts track HRV during a hard thinking task. That gives you a noninvasive way to study arousal without putting sensors all over the body.

What Is It?

Micro-expressions are brief facial changes that show up when your brain is under pressure. Think of them like a quick flicker on a screen, a small signal that can hint at strain, surprise, or effort before your face settles back to neutral.

MediaPipe helps you find facial landmarks, like the corners of your eyes, mouth, and nose. A small CNN, or convolutional neural network, learns which face patterns line up with those landmarks. HRV, or heart rate variability, is the beat-to-beat timing between heartbeats. This project asks whether the face signal and the heart signal move together during a hard thinking task.

Why This Is a Good Topic

This is a strong science fair topic because you can change one thing at a time, like task difficulty or lighting, and measure both face-based predictions and HRV. It connects to real problems in education, human factors, and mental health monitoring, where low-friction signals matter. You can learn webcam capture, feature extraction, model training, and basic statistics without needing a full wet lab.

Research Questions

• How does cognitive-load level change the frequency of detected micro-expressions?
• What is the effect of lighting changes on micro-expression classification accuracy?
• Does webcam-based arousal prediction correlate with HRV during a mental arithmetic task?
• To what extent does camera distance change MediaPipe landmark stability and model output?
• Which facial regions contribute most to arousal predictions from the CNN?
• Does a landmark-based model or a raw-frame CNN generalize better to new participants?

Basic Materials

• Laptop or desktop with a built-in or external webcam.
• External webcam with at least 1080p recording.
• Tripod or stable stand for fixed camera placement.
• Polar H10 chest strap or another HRV sensor with exportable RR intervals.
• Quiet room with steady lighting.
• Mental arithmetic or n-back task prompts.
• Consent forms, participant script, and data sheet.
• Python-capable computer with storage for video and sensor files.

Advanced Materials

• University-grade ECG or validated HRV monitor with raw RR intervals.
• Controlled lighting setup with fixed color temperature.
• High-refresh external camera with manual exposure controls.
• Quiet testing room or lab cubicle with marked seating distance.
• GPU workstation for CNN training.
• Annotation tool for facial event labeling.
• Synchronization trigger or shared timestamp logger.
• Research participant scheduling and consent materials approved by the lab.

Software & Tools

• Python: Runs video capture, feature extraction, model training, and statistics.
• MediaPipe: Detects face landmarks that become measurable coordinates for each frame.
• OpenCV: Captures webcam video and handles frame preprocessing.
• PyTorch: Trains the small CNN on labeled face clips.
• scikit-learn: Scores classification, cross-validation, and baseline models.

Experiment Steps

1. Define the arousal target you will score, such as HRV change from a baseline window.
2. Choose one high-load task and one low-load control so you can compare face signals across states.
3. Standardize camera position, lighting, and face framing so the model learns expression changes instead of scene noise.
4. Decide whether you will train on raw frames, landmark crops, or both, then build the feature pipeline around that choice.
5. Pair webcam windows with HRV windows before analysis, then lock in your accuracy, correlation, and cross-validation plan.
6. Test generalization across new participants or new lighting conditions before you finalize your conclusions.

Common Pitfalls

• Training and testing on clips from the same participant, which makes accuracy look stronger than it is on new people.
• Letting head movement vary across trials, which teaches the CNN posture changes instead of micro-expressions.
• Using a task that barely raises cognitive load, which leaves too little arousal change for validation against HRV.
• Labeling whole sessions instead of aligned time windows, which smears the face signal and the heart signal together.
• Treating eyebrow lifts, head turns, and speech movements as micro-expressions, which teaches the model the wrong facial cues.

What Makes This Competitive

A stronger project does more than report a high accuracy score. It tests subject-independent performance, compares the CNN against a simple landmark baseline, and checks whether predictions line up with HRV across time windows. If you also report failure cases under different lighting or head pose, you show that you understand the limits of the method. That makes the project feel like a real measurement study, not a one-off demo.

Project Variations

• Use the Stroop task instead of mental arithmetic, then compare whether conflict tasks produce a stronger face signal.
• Swap HRV for skin conductance, and see whether the webcam model tracks a different autonomic marker.
• Compare adult and teen participants, then test whether the same face model transfers across age groups.

Learn More

• PubMed: Search review articles on micro-expressions, cognitive load, and HRV.
• PubMed Central: Read full-text studies on facial behavior and autonomic arousal.
• PhysioNet: Find free HRV tutorials, signal definitions, and sample datasets.
• MediaPipe documentation: Learn face mesh tracking and landmark extraction.
• OpenCV documentation: Learn webcam capture, preprocessing, and video handling.