Emotion-Controlled Puppet Performance Study

ISEF Category: Technology Enhances the Arts

Subcategory: Engineering Effects  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

A puppet can say a lot without words. If its face changes with the narrator’s voice, the audience may feel the story more strongly. Your job is to test whether that actually happens. You are not just building a cool effect, you are measuring human response.

What Is It?

This project links a live voice to a soft robot puppet. A soft robot uses flexible parts instead of hard joints, so it can make smoother facial motions. Your system listens to a narrator, estimates the emotion in the speech, then moves the puppet’s face to match that guess.

Think of it like subtitles for feelings, except the subtitle is the puppet’s expression. Whisper can turn speech into text, and a small emotion classifier can sort the text or audio into labels such as happy, sad, or angry. The key question is not only whether the system works, but whether people can read the puppet’s emotion better than they can read a static puppet.

Why This Is a Good Topic

This is a strong science fair topic because you can measure both engineering performance and human perception. You can test whether a live, emotion-responsive puppet helps viewers identify mood more accurately than a puppet with one fixed face. That connects to theater, education, accessibility, and human-computer interaction. You also get real research skills, like building a prototype, designing a user study, and analyzing accuracy and confidence data.

Research Questions

• How does a voice-driven puppet affect viewers’ ability to identify the narrator’s emotion?
• What is the effect of expression timing on audience emotion accuracy?
• Does a soft-robot puppet produce higher emotion recognition than a static puppet?
• To what extent does classifier confidence predict audience-rated emotion clarity?
• Which expression style, simple facial motion or more detailed motion, leads to better audience recognition?
• How does background noise in the narrator’s voice change system accuracy?
• What is the effect of different emotion labels, such as happy, sad, and angry, on viewer agreement?

Basic Materials

• Soft-robot puppet body or foam puppet frame
• Small pneumatic actuator or air bladder system
• Mini air pump or air compressor with safe regulator
• Servo valve or solenoid valve set
• Microphone for live voice input
• Laptop or desktop computer
• USB audio interface or microphone adapter
• Speaker for playback tests
• Printed consent forms for human subjects testing
• Survey tool for audience ratings
• Camera or phone tripod for documenting trials
• Stopwatch or timing app

Advanced Materials

• Soft silicone or thermoplastic elastomer materials for facial skin
• Molds for soft facial features
• Pneumatic tubing and fittings
• Pressure sensors for actuator feedback
• Microcontroller such as Arduino or Raspberry Pi
• Valve drivers and power supply
• High-quality headset microphone
• Quiet test enclosure or acoustic panels
• Motion capture or high-speed camera for expression tracking
• Data logging interface for audio, valve state, and pressure
• User study software for randomized trial presentation
• Statistical analysis package

Software & Tools

• OpenAI Whisper: Transcribes live or recorded speech into text for emotion analysis input.
• Python: Processes audio, runs the classifier, and computes accuracy metrics.
• Audacity: Cleans audio clips and checks whether noise changes the signal.
• ImageJ: Measures puppet face motion from video frames if you track expression changes.
• Google Forms: Collects viewer ratings and confidence scores after each trial.

Experiment Steps

1. Define the emotion labels you will test and decide how the puppet will show each one.
2. Build a simple control map that converts classifier output into repeatable facial movements.
3. Set up a comparison between the responsive puppet and a static puppet with the same narrator clips.
4. Design a viewer test that measures both emotion accuracy and confidence, not just preference.
5. Plan your data analysis before you run trials, including accuracy, agreement, and effect size.
6. Check for confounds such as voice volume, speech rate, and order effects, then randomize them.

Common Pitfalls

• Training the emotion classifier on text alone, which can miss tone, pacing, and stress in the live voice.
• Using facial motions that are too subtle, so viewers cannot tell one emotion state from another.
• Letting pump delay or valve lag desynchronize the puppet face from the narrator’s voice.
• Testing the same story clips in the same order, which can create learning and fatigue effects.
• Collecting ratings without a clear scoring rubric, which makes audience accuracy hard to compare across trials.

What Makes This Competitive

A strong version of this project does more than ask whether people like the puppet. It compares different mapping rules, tests whether timing or expression intensity changes recognition, and uses a clean statistical plan. You can also separate system accuracy from audience accuracy, which often leads to smarter conclusions. If you include careful randomization, reliability checks, and a clear user study, the project starts to feel like real human-centered engineering research.

Project Variations

• Use children’s stories instead of adult narration and test whether simpler emotion labels improve audience recognition.
• Compare audio-only emotion detection with transcript-based sentiment analysis to see which input drives better puppet expressions.
• Swap the soft puppet for a screen-based avatar and compare which format gives viewers clearer emotion cues.

Learn More

• MIT OpenCourseWare Introduction to Robotics: Find courses on robot sensing, control, and human-robot interaction in the MIT OpenCourseWare catalog.
• NIH PubMed: Search review articles on emotion recognition, speech emotion detection, and human perception studies.
• NOAA Science Education: Look for basic resources on signal, noise, and data interpretation in measurement systems.
• USGS Data and Statistics: Explore examples of clean data collection, uncertainty, and graphing practices.
• Journal of Robotics and Autonomous Systems: Search peer-reviewed papers on soft robotics and expression control through a library or journal database.