Transparent Rubric Generators for Science Answers

ISEF Category: Systems Software

Subcategory: Online Learning  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

A computer can grade a science answer, but if you cannot explain why, teachers will not trust it. That is a big problem in classrooms. Your project tackles that gap by building a system that shows its rules instead of hiding them.

What Is It?

This project is about turning short science answers into clear grading rules. If a student says, "Plants need light for photosynthesis," your system tries to spot the key idea, match it to a science concept, and turn that into a rubric item the teacher can read. Think of it like a checklist built from the answer itself, not a black box score.

IE means information extraction. That is the part where software pulls useful facts out of text. Ontologies are structured maps of concepts, like a science vocabulary web that says how ideas connect. When you combine them, you can make a rubric that is easier to inspect than an AI model that just spits out a number. The goal is not only to grade, but to grade in a way that humans can audit.

Why This Is a Good Topic

This is a strong science fair topic because you can test it with real data and real metrics. You can compare your generated rubrics to teacher rubrics, then measure agreement, clarity, and error patterns. The project connects to online learning, automated grading, and classroom fairness, which are real problems schools face. You can learn natural language processing, ontology use, evaluation metrics, and software design, all in one project.

Research Questions

• How does ontology coverage affect agreement between generated rubrics and teacher rubrics?
• What is the effect of different information extraction methods on rubric accuracy?
• Does adding science concept hierarchies improve scoring agreement on short-answer responses?
• To what extent do generated rubrics match teacher scores across different question types?
• Which rubric features best predict human agreement, explicit keywords, concept links, or answer relations?
• How does transparency of rule output affect error diagnosis compared with opaque scoring?

Basic Materials

• Laptop or desktop computer with at least 8 GB RAM.
• SciEntsBank or another open short-answer science response dataset.
• A local Python install.
• Text editor or notebook environment such as Jupyter Notebook.
• Free annotation tool such as INCEpTION or doccano for marking concepts.
• Spreadsheet software for tracking rubric outputs and teacher scores.
• Access to a science concept list or ontology from a public source such as PubChem, NIH MeSH, or a relevant educational ontology.

Advanced Materials

• University workstation or cloud VM for larger text processing runs.
• GPU access if you test neural information extraction models.
• Python NLP libraries such as spaCy, scikit-learn, NLTK, and pandas.
• Ontology tools such as Protégé for exploring concept structures.
• Evaluation software or scripts for Cohen's kappa, F1, and correlation analysis.
• A version-controlled code repository for repeated experiments.
• Additional annotated corpora for transfer testing across subjects or grade levels.

Software & Tools

• Python: Builds the text pipeline, scoring logic, and evaluation scripts.
• Jupyter Notebook: Lets you inspect outputs step by step and compare rubric versions.
• spaCy: Helps extract entities, phrases, and sentence structure from student answers.
• Protégé: Lets you browse and test ontology relationships for science concepts.
• scikit-learn: Supports classification, similarity scoring, and metric calculations.

Experiment Steps

1. Define the grading target, including what counts as a correct concept, a partial credit idea, and an incorrect response.
2. Choose one science response set and one rubric style, then keep both fixed while you test your first version.
3. Map the answer text to science concepts using a transparent extraction method, not a hidden score.
4. Build a rule set that turns extracted concepts into rubric items and point decisions.
5. Compare your generated rubric scores with teacher scores using agreement metrics and error analysis.
6. Test one design change at a time, such as ontology depth, keyword rules, or answer relation handling, and compare the results.

Common Pitfalls

• Treating text similarity as understanding, which makes the system reward matching words instead of science ideas.
• Using an ontology with too many unrelated concepts, which creates false rubric matches.
• Testing only on one question type, which hides how badly the method fails on harder prompts.
• Comparing raw scores without checking agreement on partial credit, which misses the real grading problem.
• Writing rules that are hard to inspect, which breaks the transparency goal of the project.

What Makes This Competitive

A competitive version of this project would do more than build a grader. It would explain why the grader makes each decision and prove that those decisions hold up across different kinds of science questions. Strong projects also test failure cases, like paraphrases, vague answers, and multi-concept responses. If you compare multiple extraction methods and report agreement with careful statistics, your work starts to look like real research instead of a demo.

Project Variations

• Try biology short-answer questions instead of mixed science items, then check whether concept mapping works better in one subject.
• Replace rule-based extraction with a hybrid method that uses rules first and a lightweight classifier second.
• Test whether your rubric generator works better on simple fact recall or on explanations that require cause and effect.

Learn More

• SciEntsBank corpus papers: Search Google Scholar or PubMed-linked references for studies on automated scoring of short science answers.
• National Library of Medicine MeSH: Use the Medical Subject Headings browser to explore concept hierarchies and related terms.
• NIH PubMed: Search for review articles on automated short-answer grading, information extraction, and educational NLP.
• Stanford NLP resources: Use university course materials and reading lists on natural language processing and text classification.
• Protégé documentation: Learn how ontologies are structured and edited by searching for the official Protégé user guide.
• scikit-learn user guide: Find the official documentation for model evaluation, classification, and metric calculations.