Storybook AI for Better Reading Comprehension

ISEF Category: Technology Enhances the Arts

Subcategory: Human Information Exchange  ·  Difficulty: Advanced  ·  Setup: School Lab  ·  Time: Full Year

The Hook

A story can feel flat if the reader loses the room. Your project tries to fix that by letting the book react to the listener. If a child looks confused or bored, the system can explain a word or add a sound cue. That makes this a real test of whether adaptive storytelling helps comprehension.

What Is It?

This project mixes three pieces of technology. OCR, or optical character recognition, turns printed text into digital text. TTS, or text-to-speech, reads that text aloud. FaceMesh tracks facial landmarks, which are tiny points around the eyes, mouth, and face shape, so the system can guess when a listener looks confused or bored.

Think of it like a smart audiobook with a built-in sidekick. A normal audiobook just plays from start to finish. Your version watches for signs that the listener may need help, then changes the story flow in response. That could mean pausing to define a word, repeating a sentence, or adding a sound effect that keeps attention up.

Why This Is a Good Topic

This is a strong science fair topic because you can test it with a clear comparison. You can compare a static read-aloud version against an adaptive version and measure comprehension, recall, or engagement. The project connects to education tech, accessibility, and human-computer interaction, all of which matter in real classrooms and libraries. You can also learn computer vision, speech systems, experiment design, and statistics.

Research Questions

• How does adaptive narration affect reading comprehension scores compared with a fixed read-aloud version?
• What is the effect of vocabulary explanations on recall of new words in a picture book?
• Does adding sound effects increase attention or memory for story events?
• To what extent can FaceMesh-based confusion signals predict moments when a reader misses key content?
• Which adaptation strategy, vocabulary help, repetition, or sound cues, produces the highest comprehension gain?
• How does the system perform across different book genres, such as fiction and nonfiction?

Basic Materials

• Laptop or desktop computer with webcam.
• Printed picture books with age-appropriate text.
• Microphone and speakers or headphones.
• Python installed on the computer.
• Webcam stand or stable mount.
• Consent forms and parent or guardian permission for any child participants.
• Spreadsheet software for recording scores and observations.
• Basic survey or quiz forms for comprehension testing.

Advanced Materials

• Higher-quality webcam with good low-light performance.
• External microphone for cleaner speech capture.
• Tablet or second monitor for participant-facing prompts.
• Python libraries for OCR, face landmark detection, and text-to-speech.
• Annotated video clips for training or checking facial expression labels.
• Statistical analysis software or Python notebooks.
• Secure data storage for participant recordings and scores.
• Optional eye-tracking or behavior coding software if available in a university lab.

Software & Tools

• Python: Runs the OCR, face tracking, narration logic, and data analysis scripts.
• MediaPipe: Detects face landmarks and supports expression-based attention analysis.
• Tesseract OCR: Converts printed book pages into machine-readable text.
• pyttsx3: Plays synthetic speech for a fully local text-to-speech setup.
• ImageJ: Helps inspect screenshots or frame captures when you need to compare facial landmark placement.

Experiment Steps

1. Define the reading problem you want to solve, such as missed vocabulary, low recall, or loss of attention.
2. Choose one adaptation rule first, then decide what face signal or user behavior will trigger it.
3. Build a simple baseline version that reads the book the same way every time.
4. Add one adaptive feature, then plan a fair A/B comparison against the baseline.
5. Design your comprehension measures so they match the book and the age group you test.
6. Plan how you will score results, compare groups, and check whether the adaptive system really helps.

Common Pitfalls

• Using a face signal as if it proves confusion, when it may only show a neutral expression or normal blinking.
• Testing with books that OCR reads poorly, which causes narration errors that hide the real effect of adaptation.
• Changing several features at once, which makes you unable to tell whether vocabulary help or sound effects caused the result.
• Measuring only enjoyment, which does not show whether the system improved comprehension or recall.
• Ignoring privacy and consent for child participants, which can stop the project before data collection begins.

What Makes This Competitive

A stronger version of this project goes beyond a simple demo. You would need a clean baseline, a clear adaptive rule, and a careful A/B test with enough participants to compare outcomes fairly. Strong entries also test more than one kind of adaptation and analyze where the system helps or fails. If you can connect facial signals to comprehension gains with thoughtful statistics, the project looks much closer to real research.

Project Variations

• Test the system with bilingual storybooks to see whether adaptive vocabulary help works better for second-language readers.
• Replace face tracking with simple button taps or voice prompts, then compare which input method produces cleaner adaptation decisions.
• Measure whether the same adaptive narration helps younger children more than older readers by comparing age groups.

Learn More

• MediaPipe documentation: Read the official face mesh guide and examples on the MediaPipe site.
• Tesseract OCR documentation: Learn how open-source OCR works from the official Tesseract project pages.
• NIH PubMed: Search for review articles on child reading comprehension, adaptive tutoring, and human-computer interaction.
• ERIC: Find education research on read-alouds, vocabulary support, and comprehension assessment in classrooms.
• MIT OpenCourseWare: Look for free computer vision and machine learning course materials that explain image features and classification.
• IEEE Xplore or ACM Digital Library: Search for peer-reviewed papers on adaptive educational interfaces and affective computing.