Accent-Preserving Dubbing for Indie Films

ISEF Category: Technology Enhances the Arts

Subcategory: Human Information Exchange  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

What if you could dub a film into another language without losing the actor's voice? That is the promise of this kind of audio pipeline. You are not just changing words. You are trying to keep timbre, accent, and emotional style intact while the language changes.

What Is It?

This project sits at the intersection of speech processing and filmmaking. In simple terms, you take an audio track, separate the voice from the background, change the spoken language, then rebuild the voice so it still sounds like the same person. Demucs can help isolate the vocal track, MarianMT can translate the words, and RVC can resynthesize speech in a target voice style. The hard part is keeping the result natural, not robotic.

Think of it like re-dubbing a movie with a careful impressionist. The words change, but the voice should still feel like the same actor. In this project, you are studying how well a pipeline preserves speaker identity, accent character, and speech quality after translation and voice conversion. You can measure that with speaker-verification cosine similarity, plus human ratings or speech quality scores.

Why This Is a Good Topic

This makes a strong science fair topic because you can test real engineering choices and measure the results. You can compare different translation models, voice-conversion settings, and source audio types, then ask which setup best preserves identity and clarity. The project connects to film localization, accessibility, and multilingual media, so the real-world use case is clear. You can also learn speech processing, machine learning evaluation, and experimental design.

Research Questions

• How does the choice of source audio quality affect speaker-verification cosine similarity after dubbing?
• What is the effect of different noise levels in the input track on the final voice identity score?
• Does using accent-preserving voice conversion produce higher listener ratings than standard voice conversion?
• To what extent does the language pair change the balance between translation accuracy and voice similarity?
• Which separation method gives the cleanest vocal stem for downstream dubbing?
• How does the amount of background music in the original clip affect speech quality in the translated output?

Basic Materials

• A computer with a modern GPU or cloud access for inference.
• A set of short video or audio clips with clear speaker identity.
• Headphones for careful listening tests.
• Open-source speech separation software such as Demucs.
• Open-source text translation software such as MarianMT.
• Open-source voice conversion software such as RVC.
• Speaker verification model or embedding extractor for similarity scoring.
• Spreadsheet software for logging scores and trial conditions.
• Audio editor such as Audacity for quick inspection of clips.

Advanced Materials

• A workstation with a CUDA-capable GPU and enough memory for audio models.
• A labeled multilingual speech dataset for benchmarking.
• A forced-alignment tool for checking word timing after translation.
• A speech-to-text system for transcript comparison.
• A speaker embedding model such as ECAPA-TDNN for identity scoring.
• A phonetic analysis toolkit for accent feature comparison.
• A small listening panel and survey platform for human ratings.
• Python environment with audio processing libraries.

Software & Tools

• Python: Runs the pipeline, logs experiments, and computes similarity metrics.
• Demucs: Separates vocals from background music and effects.
• MarianMT: Translates text between languages using open models.
• RVC: Converts the translated speech into a target voice style.
• ImageJ: Not used here, so skip this tool and choose an audio workflow instead.

Experiment Steps

1. Define the exact audio problem you want to solve, such as language switching with identity preservation or accent retention.
2. Choose one evaluation target first, then decide whether you will score speaker similarity, intelligibility, or both.
3. Select a small set of source clips that vary in noise, accent strength, or background music so you can compare conditions fairly.
4. Build one baseline pipeline, then plan a second version with a single changed component so you can isolate the effect.
5. Design your scoring method before you run the full batch, including automated similarity metrics and a simple human rating form.
6. Plan controls that separate translation errors from voice-conversion errors, so you know which stage caused each result.

Common Pitfalls

• Using clips with inconsistent microphone quality, which makes voice similarity scores hard to compare.
• Mixing translation changes with voice conversion changes in the same test, which hides the source of any improvement or drop.
• Relying only on the model output sound, which can miss errors in transcript meaning or accent drift.
• Comparing clips without matching duration or speaking style, which can bias speaker-verification scores.
• Forgetting to clean background music and noise first, which can confuse both the translation stage and the voice model.

What Makes This Competitive

A stronger project will not just run the pipeline once. It will compare multiple pipeline choices, define clear metrics, and test whether the results hold across different accents, languages, and audio conditions. You can push the project further by separating subjective listening scores from automatic similarity scores, then checking where they agree or disagree. That kind of careful analysis makes the work feel like real research, not just a demo.

Project Variations

• Test the pipeline on animated short clips instead of live-action dialogue to see whether stylized speech changes identity preservation.
• Compare accent retention across two language pairs, then measure whether phonetic distance affects the final voice score.
• Replace the speaker-verification metric with listener surveys and analyze how human perception differs from model-based scoring.

Learn More

• PubMed: Search for review articles on speech synthesis, speaker verification, and voice conversion to understand common evaluation methods.
• NIH 3D Print Exchange is not relevant here, so use NIH PubMed Central instead: Search for free full-text papers on neural speech processing and multilingual TTS.
• arXiv: Search for preprints on speech translation, voice conversion, Demucs, and speaker embeddings.
• MIT OpenCourseWare: Look for free computer science and signal processing courses that cover audio feature extraction and machine learning basics.
• NASA Open Science data resources are not relevant here, so use the National Institute of Standards and Technology speech resources instead: Search for published evaluation guidance on speech and biometric similarity metrics.