Socratic AI Tutor Guardrails

ISEF Category: Systems Software

Subcategory: Online Learning  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

A chatbot can give a right answer in one second, but that does not mean you learned anything. If you build a tutor that refuses to solve the problem too soon, you may help students think harder. The tricky part is making that refusal feel useful, not annoying. That is where your research starts.

What Is It?

This project studies a kind of middleware, which is software that sits between a user and an AI model. Your guardrail checks the student’s message, then decides whether to give a direct answer or to push with hints and questions first. Think of it like a coach who asks, “What have you tried?” before handing over the next clue.

That idea matters because many tutoring chatbots are too eager. They can turn into answer machines. A Socratic guardrail tries to protect learning by steering the conversation back to reasoning, not copying. In your project, you would test whether that design improves algebra learning, and whether students can bypass it with jailbreak prompts that try to force direct answers.

Why This Is a Good Topic

This is a strong science fair topic because you can test it with real users, clear metrics, and real software design choices. You can measure learning gains, response helpfulness, and jailbreak resistance, so the project is more than a demo. It also connects to a real problem, since schools are trying to use AI without replacing student thinking. You can realistically learn about prompt design, experiment design, usability, and basic statistics while building something original.

Research Questions

• How does a Socratic guardrail change post-test algebra scores compared with a direct-answer tutor?
• What is the effect of requiring a student attempt before giving help on solution quality?
• Does a guardrail reduce the number of fully solved answers the model gives on first turn?
• To what extent does the guardrail resist common jailbreak prompts that ask for direct answers?
• Which hint style, question-first or error-focused, leads to better student revision after an incorrect attempt?
• What is the effect of different refusal messages on student persistence and frustration?

Basic Materials

• Laptop or desktop computer with internet access.
• Access to an LLM API or web-based model with prompt control.
• Spreadsheet software for logging responses and scores.
• Small set of algebra problems at one difficulty level.
• Pre-test and post-test forms.
• Consent and assent forms if you test with students.
• Simple survey for perceived helpfulness and frustration.
• Screen recording software for response review.

Advanced Materials

• Server or local middleware environment for routing model outputs.
• API access logs with timestamps and prompt versions.
• Annotation tool for labeling student attempts and model response types.
• Statistical software such as R or Python.
• Item-response or rubric-based scoring sheets for algebra solutions.
• Jailbreak prompt test suite with categories of attack style.
• Student interface prototype for controlled user testing.
• Secure database for storing anonymized session data.

Software & Tools

• Python: Organizes prompt tests, scores outputs, and runs basic statistics.
• R: Fits comparison models and visualizes learning gains across conditions.
• Google Sheets: Tracks prompts, ratings, and pre-test or post-test results.
• ImageJ: Not needed for this topic, so skip it unless you add screenshot-based analysis.
• Qualtrics: Collects survey answers and student feedback in a structured way.

Experiment Steps

1. Define the exact behavior your guardrail should block, such as direct answers before an attempt.
2. Choose one tutoring task, like algebra word problems, so you can compare conditions fairly.
3. Build two or more chatbot versions, one with the guardrail and one without it.
4. Design a scoring plan for learning gain, response quality, and jailbreak resistance.
5. Pilot the system with a small group and fix any confusion in the prompts or interface.
6. Run the main study, then compare how each version affects student performance and model behavior.

Common Pitfalls

• Writing a guardrail that blocks too much, which makes the tutor useless and noisy.
• Testing on mixed algebra topics, which hides whether the middleware helps on a specific skill.
• Measuring only chatbot politeness, which misses whether students actually learned more.
• Using jailbreak prompts that are too fake or too easy, which gives a false picture of security.
• Letting the model see different problem formats across conditions, which makes the learning comparison unfair.

What Makes This Competitive

A strong version of this project would compare more than one guardrail design and measure both learning and resistance to prompt attacks. You could separate student attempt quality from final score gains, then test whether certain hint patterns work better for different learners. A serious entry would also use clean controls, preregistered scoring rules, and statistics that match repeated measures data. That gives your project a real systems-software angle, not just a chatbot demo.

Project Variations

• Test the guardrail on geometry or chemistry word problems instead of algebra to see whether the same design generalizes.
• Compare a strict refusal policy with a softer hint ladder to measure which one keeps students engaged longer.
• Analyze jailbreak resistance using prompt categories such as roleplay, instruction override, and hidden answer requests.

Learn More

• MIT OpenCourseWare: Search for courses on software engineering, human-computer interaction, and machine learning to study system design and user testing.
• Stanford CS229 course notes: Search the public course materials for machine learning basics that help you think about model behavior.
• NIH PubMed: Search for review articles on intelligent tutoring systems, educational chatbots, and student learning outcomes.
• arXiv: Search for recent preprints on LLM guardrails, prompt injection, and educational AI systems.
• ACM Digital Library: Search for peer-reviewed papers on tutoring systems, dialogue systems, and usability evaluation.
• OpenAI Cookbook: Read the free examples on prompt design, evaluation, and API-based workflows.