Predicting Explanatory Depth with AI

ISEF Category: Behavioral and Social Sciences

Subcategory: Cognitive Psychology  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

You can feel sure you understand a concept until someone asks you to explain it step by step. Then the gaps show up fast. This project turns that common brain glitch into a prediction problem, using language models to guess which everyday ideas will expose the biggest confidence drop.

What Is It?

The illusion of explanatory depth happens when you think you understand something better than you really do. A light switch, a zipper, or a voting system can feel obvious until you try to explain every step. The act of explaining forces your brain to reveal the missing links.

In this project, you build a model that looks at features of a concept, such as familiarity, concreteness, or domain, and predicts how much a person's self-rating will fall after they try to explain it. Think of it like a forecast for overconfidence. The goal is not to guess whether people know the answer. The goal is to guess where self-assurance will shrink the most once explanation starts.

Why This Is a Good Topic

This topic works well because you can measure it with surveys, short explanations, and simple text features, so you do not need a wet lab. It connects to metacognition, confidence, and how people judge their own knowledge, which matters in classrooms, tutoring, and product design. You can learn how to turn messy human judgments into a prediction task, clean the data, and test whether your model does better than a simple baseline.

Research Questions

• How does concept familiarity affect the size of the self-rated understanding drop after forced explanation?
• What is the effect of concept concreteness on how much confidence changes after explanation?
• Does a fine-tuned small language model predict the understanding drop better than a baseline model that uses only word length and frequency?
• To what extent do domain labels, such as science, technology, and everyday life, improve prediction accuracy?
• Which feature set, familiarity, concreteness, or part-of-speech cues, best ranks concepts by expected confidence drop?
• How does participant prior knowledge change the gap between initial confidence and post-explanation ratings?

Basic Materials

• Laptop with internet access.
• Google Forms or paper survey sheets.
• Spreadsheet software such as Google Sheets or Excel.
• A list of 30 to 60 everyday concepts to test.
• Printed instruction sheet for participants.
• Informed consent and assent forms.

Advanced Materials

• University-approved survey platform such as Qualtrics or REDCap.
• GPU-capable computer for fine-tuning.
• Small language model checkpoints from Hugging Face.
• Python environment with PyTorch, transformers, pandas, and scikit-learn.
• R, JASP, or SPSS for statistical analysis.
• Codebook for concept features and participant ratings.

Software & Tools

• Python: Cleans survey data, builds features, and trains baseline models.
• PyTorch: Fine-tunes a small language model on concept-level prediction labels.
• scikit-learn: Compares simple baselines and scores classification or ranking performance.
• R: Runs correlations, regression models, and effect size calculations.
• JASP: Lets you check t tests, ANOVA, and assumption plots without paid software.

Experiment Steps

1. Define the outcome you want to predict, such as the post-explanation drop in self-rating.
2. Build a concept list that varies in familiarity, concreteness, and domain.
3. Decide how participants will rate confidence before and after forced explanation.
4. Split your data into training and test sets, then compare a simple baseline with the fine-tuned model.
5. Choose metrics that reward correct ranking of the hardest concepts, not just average accuracy.

Common Pitfalls

• Asking participants to explain concepts that are too technical or too obscure, which turns the study into a knowledge test instead of an explanatory-depth test.
• Mixing up confidence before explanation and confidence after explanation, which makes the drop score unreliable.
• Training on the same concepts you later report, which inflates model performance and hides overfitting.
• Using concept lists with uneven topic balance, which lets one domain drive the results.
• Letting participants copy a memorized definition before the second rating, which removes the self-generated gap you want to measure.

What Makes This Competitive

A stronger entry would not just predict the drop, it would explain why the model is right. You could compare a fine-tuned language model with simpler psychology features such as familiarity, concreteness, and prior confidence. You could also test whether the model ranks the hardest concepts better than chance across different participant groups. A clear error analysis, with attention to which concepts the model misses, would make the project feel much more serious.

Project Variations

• Compare teen participants with adults to see whether age changes the size of the explanatory-depth drop.
• Swap everyday concepts for science concepts to test whether familiarity or abstractness matters more.
• Use only text features such as word length, frequency, and concreteness, then compare that baseline with the fine-tuned model.

Learn More

• PubMed: Search for review articles on the illusion of explanatory depth, metacognition, and confidence calibration.
• PubMed Central: Read free full-text papers on self-explanation, overconfidence, and concept knowledge.
• NCBI Bookshelf: Find free background chapters on cognition, judgment, and reasoning.
• MIT OpenCourseWare: Look for introductory psychology and cognition lecture notes and readings.
• JASP: Read the free documentation for t tests, regression, and effect size analysis.
• Hugging Face Transformers documentation: Learn how to fine-tune small language models and run text baselines.