Decoding Imagined vs. Heard Speech in fMRI

ISEF Category: Computational Biology and Bioinformatics

Subcategory: Computational Neuroscience  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

Your brain can hear a sentence without sound. That makes imagined speech a weird and useful test case for AI. If a model can tell imagined speech from perceived speech in fMRI, you can ask where the two signals split. That gives you a real neuroscience question and a real machine learning problem in one project.

What Is It?

This project uses fMRI, which stands for functional magnetic resonance imaging, to study two kinds of speech processing. One happens when you hear words. The other happens when you only imagine them. fMRI does not read thoughts directly. It measures changes in blood flow that track brain activity, like a heat map for neural work.

A transformer model is a type of machine learning model that looks for patterns across many inputs at once. In this project, the inputs are brain scans from public datasets such as OpenNeuro. The model learns to separate imagined speech from perceived speech. Then you inspect attention maps, which show which parts of the input the model relied on most. That can help you compare the brain patterns behind inner speech and heard speech.

Why This Is a Good Topic

This is a strong science fair topic because you can ask a clear yes-or-no classification question, then go deeper with interpretation. You get to work with real human data, a public dataset, and modern AI methods. The project connects to speech disorders, brain-computer interfaces, and cognitive neuroscience. You can learn data cleaning, feature extraction, model evaluation, and scientific reasoning without needing to run your own MRI scanner.

Research Questions

• How does a transformer model perform when classifying imagined speech versus perceived speech from fMRI data?
• What is the effect of using different brain regions as input on classification accuracy?
• Does attention-map analysis identify consistent areas that separate imagined speech from perceived speech?
• To what extent does subject-wise cross-validation change the reported model performance?
• Which preprocessing choice, such as smoothing or normalization, most changes the model's ability to separate the two speech conditions?
• How does model performance compare between a transformer and a simpler baseline classifier on the same features?
• What is the effect of training on one public fMRI dataset and testing on another on generalization?

Basic Materials

• Computer with enough RAM to handle fMRI files and model training.
• Public OpenNeuro dataset with imagined and perceived speech fMRI scans.
• Python installed with scientific computing libraries.
• Jupyter Notebook or VS Code for analysis.
• NumPy for array handling.
• Pandas for tabular metadata.
• scikit-learn for baseline models and evaluation.
• PyTorch or TensorFlow for transformer modeling.
• Nibabel for loading neuroimaging files.
• nilearn for fMRI preprocessing and visualization.
• Matplotlib for plots and figures.
• External storage or cloud space for large dataset files.

Advanced Materials

• Access to a university or school server with a GPU.
• High-memory workstation for training and cross-validation.
• FSL or SPM for advanced fMRI preprocessing.
• AFNI for alternate neuroimaging analysis workflows.
• MNE-Python for signal handling if you compare modalities.
• Nipype for reproducible pipeline management.
• BrainSpace or Connectome Workbench for brain mapping visualizations.
• Docker or Singularity for environment control.

Software & Tools

• OpenNeuro: Hosts public neuroimaging datasets you can search for imagined and perceived speech tasks.
• Python: Lets you clean data, train models, and run analysis scripts.
• nilearn: Helps you load, preprocess, and visualize fMRI data in Python.
• PyTorch: Supports transformer models and attention-map inspection.
• scikit-learn: Provides baseline classifiers, cross-validation, and metrics for comparison.

Experiment Steps

1. Define the exact speech contrast you will test and the subject groups you will include.
2. Choose one preprocessing pipeline and one baseline feature set before training any model.
3. Build a fair evaluation plan that keeps subjects separated across train, validation, and test splits.
4. Train a baseline classifier first, then compare it with the transformer on the same data.
5. Inspect attention maps and test whether they point to stable brain regions across subjects.
6. Plan a second analysis that checks whether your result holds across datasets, regions, or preprocessing choices.

Common Pitfalls

• Mixing scans from the same subject across train and test splits, which inflates accuracy without real generalization.
• Using all voxels without dimensionality reduction, which can swamp the model with noise and memory limits.
• Comparing models on different preprocessing pipelines, which makes the accuracy numbers unfair.
• Treating attention weights as direct proof of brain causation, which overstates what the model can tell you.
• Ignoring class balance between imagined and perceived speech, which can make a weak model look strong.

What Makes This Competitive

A stronger project goes past simple classification. You need clean subject-wise validation, a clear baseline, and a careful test of whether the model generalizes across people or datasets. The best versions also compare attention maps with other interpretability methods, so you do not rely on one signal alone. If you can connect model behavior to a real neurobiological question about speech imagery, your project becomes much more than a coding exercise.

Project Variations

• Use motor imagery instead of speech imagery and compare how well the same transformer generalizes across tasks.
• Test whether a region-limited model from auditory cortex, language cortex, or frontal cortex separates the two conditions better.
• Compare transformer attention maps with simpler saliency methods to see whether the same brain regions appear in both analyses.

Learn More

• OpenNeuro: Search for public fMRI datasets with speech imagery and speech perception tasks.
• NIH PubMed: Search for review articles on inner speech, speech perception, and fMRI decoding.
• NIH Brain Initiative: Find background on human brain mapping and neuroimaging methods.
• MIT OpenCourseWare: Search for courses on machine learning and deep learning for model-building basics.
• nilearn documentation: Read examples for loading, preprocessing, and plotting neuroimaging data.
• Nature Neuroscience and NeuroImage: Search for peer-reviewed papers on speech decoding, attention maps, and fMRI classification.