Fairness Dashboards for ML Model Testing

ISEF Category: Systems Software

Subcategory: Human/Machine Interface  ·  Difficulty: Advanced  ·  Setup: School Lab  ·  Time: 1 to 2 Months

The Hook

A model can look fair on paper and still fail one group in practice. That gap hides in the details, like a microscope that looks sharp until you move the sample. Your project can help people spot that gap faster. You can build a dashboard that makes model behavior easier to probe across demographic slices.

What Is It?

A fairness dashboard is an interactive screen that helps people test how a machine learning model behaves for different groups. Think of it like a car dashboard. You do not need to know how the engine works to see the speed, fuel level, and warning lights. In the same way, a fairness dashboard helps a user inspect prediction scores, errors, and group differences without reading the model code.

The direct manipulation part means the user can change inputs and see results right away. They might adjust age, education, or work hours and watch the prediction change. That makes the model easier to question. Instead of trusting one summary number, users can compare slices, or subgroups, and look for patterns like one group getting more false positives than another.

Why This Is a Good Topic

This is a strong science fair topic because you can test both the interface and the analysis. You can measure whether people find group differences faster, understand model behavior better, or make fewer mistakes when they use your dashboard instead of a standard tool. The topic connects to hiring, lending, health, and any system that uses ML to affect people. You can learn UI design, basic fairness metrics, and usability testing without needing a wet lab.

Research Questions

• How does a fairness dashboard change how quickly users find performance gaps across demographic slices?
• What is the effect of direct manipulation controls on user accuracy when comparing model outcomes for different groups?
• Does a fairness dashboard help non-ML users identify false positive and false negative disparities more accurately than a static report?
• To what extent does showing multiple fairness metrics at once improve users' understanding of model behavior?
• Which dashboard layout helps users complete group comparison tasks with the fewest errors?
• How does prior ML experience affect how users interpret fairness results in an interactive dashboard?

Basic Materials

• Laptop or desktop computer with a modern web browser.
• Python 3 with Jupyter Notebook or Google Colab access.
• Public dataset such as the UCI Adult dataset.
• Open-source ML library such as scikit-learn.
• Simple web app framework such as Streamlit or Plotly Dash.
• Survey form tool such as Google Forms for usability testing.
• Spreadsheet software for logging task times and survey answers.

Advanced Materials

• University or school server for repeated user testing sessions.
• Version-controlled code repository with issue tracking.
• Eye-tracking system or screen-recording software for interaction analysis.
• Statistical analysis package such as R or Python with SciPy and statsmodels.
• Accessibility testing tools for color contrast and keyboard navigation.
• Secure storage for user study data and consent forms.

Software & Tools

• Streamlit: Builds an interactive dashboard quickly and lets you test direct manipulation ideas.
• Plotly Dash: Creates data-driven web interfaces for exploring model outputs and slices.
• Python: Prepares the dataset, trains the baseline model, and computes fairness metrics.
• scikit-learn: Trains a simple reference model for your comparison study.
• Google Forms: Collects user feedback, task success, and confidence ratings.

Experiment Steps

1. Define the user problem you want to solve, such as helping non-ML users compare fairness across groups.
2. Pick one baseline model and the slice variables you will let users explore.
3. Design the dashboard interactions, then decide which fairness metrics and visual cues users will see.
4. Plan a usability test that compares your dashboard against a simpler reference view.
5. Choose outcome measures, such as task accuracy, completion time, confidence, and error rate.
6. Set up your analysis plan before recruiting users, so you know how you will compare results.

Common Pitfalls

• Treating fairness as one number, which hides different error patterns across groups.
• Using a dataset with messy labels or missing values, which makes the dashboard results hard to trust.
• Packing too many charts into one screen, which overloads first-time users and slows task completion.
• Testing only one user group, which makes the usability results too narrow to support strong claims.
• Comparing your dashboard to What-If Tool without matching the tasks, which makes the study unfair.

What Makes This Competitive

A stronger project goes beyond building a pretty interface. You can compare your dashboard with a known tool using matched tasks, the same dataset, and clear success metrics. You can also test whether users with no ML background understand fairness better when you change the layout, the labels, or the way slices are grouped. Strong analysis, careful controls, and a clear user study design can turn this into a serious systems software project.

Project Variations

• Use a different dataset with hiring or lending features, then test whether the dashboard still helps users find group gaps.
• Swap the fairness metrics you display, such as comparing accuracy, false positive rate, and calibration across slices.
• Focus on accessibility by testing whether color, labels, and keyboard controls help novice users interpret the dashboard faster.

Learn More

• Google What-If Tool documentation: Read about interactive model inspection and fairness analysis in the official TensorFlow tools docs.
• UCI Machine Learning Repository: Find the Adult dataset and other public datasets for fair ML testing.
• NIH PubMed: Search for review articles on algorithmic fairness, explainable ML, and human factors in decision support.
• NIST AI Risk Management Framework: Read guidance on measuring and managing AI risks, including fairness and usability concerns.
• MIT OpenCourseWare: Look for free courses on human-computer interaction, data visualization, and machine learning fundamentals.