Early-Warning LMS Dashboard With Fairness Checks

ISEF Category: Systems Software

Subcategory: Online Learning  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

A dashboard can look smart and still be unfair. One student may get flagged early, while another slips by because the model reads their click pattern wrong. That makes this topic more than a coding project. You are building a system that could change how teachers act.

What Is It?

This phenomenon asks whether a learning platform can predict, early enough to matter, which students are likely to struggle in the next 2 weeks. The model uses clickstream data, which means patterns like logins, page views, assignment visits, and gaps in activity. Think of it like a weather forecast for class performance. The goal is not to guess perfectly, but to give teachers a useful heads-up with a clear confidence level.

The fairness part matters because models can behave differently across student groups even when they never see demographics during training. In the OULAD dataset, you can test whether the model treats subgroups more or less accurately. Calibrated uncertainty means the model should know when it is unsure, not just output a yes-or-no answer. That makes the system safer and more useful.

Why This Is a Good Topic

This is a strong science fair topic because you can test a real prediction system with public data, measurable metrics, and clear design choices. You can compare models, tune thresholds, and measure fairness with numbers instead of opinions. The project connects to a real problem in online learning, which is spotting struggling students before they fail. You can also learn data cleaning, feature engineering, classification, calibration, and bias checks.

Research Questions

• How does adding recent clickstream recency features change next-2-week risk prediction accuracy?
• What is the effect of calibration methods on the reliability of predicted student risk scores?
• Does a model trained only on LMS behavior show different error rates across OULAD subgroups?
• To what extent do activity gaps versus total activity counts improve early-warning prediction?
• Which feature set gives the best balance between precision, recall, and calibration?
• How does changing the decision threshold affect false alarms for low-risk students?

Basic Materials

• Laptop or desktop computer with at least 8 GB RAM.
• Public OULAD dataset or another LMS clickstream dataset.
• Python installed with Jupyter Notebook.
• Pandas for data cleaning and feature building.
• Scikit-learn for classification and evaluation.
• Matplotlib or Seaborn for plots.
• Spreadsheet software for quick checks and labeling.

Advanced Materials

• University or high-end laptop workstation with 16 GB RAM or more.
• Python environment with scikit-learn, XGBoost or LightGBM, and calibration tools.
• Jupyter Notebook or JupyterLab.
• Git for version control.
• Database or parquet storage for larger feature tables.
• Fairness analysis package such as Fairlearn or AIF360.
• Visualization tools for reliability curves and subgroup metrics.

Software & Tools

• Python: Cleans clickstream data, builds features, and trains prediction models.
• Jupyter Notebook: Lets you test ideas and document each modeling step in one place.
• scikit-learn: Fits classifiers, checks cross-validation, and measures calibration.
• Pandas: Organizes LMS records into student-level and week-level tables.
• Fairlearn: Compares error rates and performance across student subgroups.

Experiment Steps

1. Define what counts as falling behind in the next 2 weeks, and turn that into a clear label.
2. Build student-level features from clickstream history, then decide which time window gives the fairest forecast.
3. Split the data by student and by course period, so your test set reflects a real early-warning setting.
4. Train a baseline classifier, then compare it with a stronger model and a calibrated version.
5. Check reliability plots, confusion matrices, and subgroup metrics to see where the system helps and where it fails.
6. Choose a decision threshold that balances false alarms, missed warnings, and fairness across groups.

Common Pitfalls

• Using random row splits instead of student-level splits, which leaks the same student into both training and test data.
• Building features from the full course timeline, which lets future behavior sneak into the prediction.
• Treating high accuracy as success, even when the model misses most struggling students.
• Ignoring calibration, which makes risk scores look confident when they should be uncertain.
• Reporting only overall accuracy and not subgroup error rates, which hides unfair behavior across OULAD groups.

What Makes This Competitive

A competitive version of this project goes beyond one model and one metric. You compare several feature windows, test calibration, and measure subgroup behavior with more than simple accuracy. Strong projects also explain tradeoffs, like whether the model becomes more fair when it becomes less sensitive. If you can show a clearer, safer decision rule than a standard baseline, your work starts to look like real systems research.

Project Variations

• Use a different definition of early warning, such as missing two assignments instead of falling behind overall.
• Swap in a different model family, such as gradient boosting, logistic regression, or a small neural net, and compare fairness results.
• Test whether removing recency features and using only cumulative behavior changes calibration and subgroup error patterns.

Learn More

• OULAD dataset page: Find the Open University Learning Analytics Dataset and its documentation by searching the dataset name online.
• Proceedings of the ACM on Human-Computer Interaction: Search for papers on learning analytics, early warning systems, and fairness in educational data.
• PubMed: Search for review articles on educational data mining, student risk prediction, and bias in algorithmic decision support.
• MIT OpenCourseWare: Look for free machine learning and statistics course materials to strengthen model evaluation skills.
• NIH Office of Data Science Strategy: Read free material on data quality, reproducibility, and responsible data analysis.