LLM Agent Reliability Benchmark

ISEF Category: Systems Software

Subcategory: Other  ·  Difficulty: Advanced  ·  Setup: School Lab  ·  Time: Full Year

The Hook

A chatbot can sound smart and still fail the task. That gap matters when an AI has to run commands, edit files, and keep going for dozens of steps. If you can measure that gap, you can build a benchmark that tells people which models actually work. That kind of metric is gold in AI system testing.

What Is It?

This project asks a simple question with a hard answer, how reliable is an AI agent when it has to use tools over a long task? Think of the model as a student with a laptop, a terminal, and a messy to-do list. A short answer can look great in one chat window. A long task tests something else, like whether the model keeps track of state, follows commands, and recovers from small mistakes.

A benchmark is just a set of tasks with a scoring rule. In this case, the tasks live in a Docker sandbox, which is a sealed computer environment that always starts the same way. That matters because you want fair tests. If one model gets a lucky setup and another does not, your score stops meaning much. Your job is to design or refine the metric, then see whether it really separates models that just talk well from models that can act well.

Why This Is a Good Topic

This is a strong science fair topic because you can test it, measure it, and compare systems in a way that other people can repeat. The real-world problem is AI reliability, which matters for coding assistants, automation tools, and agent systems that take actions instead of just answering questions. You can learn benchmark design, reproducibility, evaluation, and basic experimental statistics. You also get to work on a problem that has real research value, not just a demo.

Research Questions

• How does the new reliability metric compare with final-task-success alone for ranking LLM agents on long-horizon CLI tasks?
• What is the effect of task length on agentic-tool-use reliability scores?
• Does reproducible Docker setup reduce score variance across repeated runs?
• To what extent do different scoring scripts change the leaderboard ranking of the same models?
• Which task types, such as file editing, command chaining, or error recovery, produce the largest reliability drop?
• How does adding intermediate-state checks affect the metric's ability to detect fragile agent behavior?
• What is the effect of prompt structure on the consistency of agent tool use across benchmark tasks?

Basic Materials

• A laptop or desktop computer with enough storage to run containers and scripts.
• Docker Desktop or Docker Engine for creating a reproducible sandbox.
• Python 3.10 or later for running evaluation scripts and data analysis.
• A code editor such as Visual Studio Code for reading and editing benchmark files.
• Git for tracking changes and comparing versions of the benchmark.
• A spreadsheet or notebook for logging runs, scores, and observations.
• A small set of open-source CLI tasks or a SWE-bench-style task subset for pilot testing.

Advanced Materials

• A workstation with more CPU cores and memory for repeated benchmark runs.
• A local GPU or access to a shared compute server for testing larger models or faster inference setups.
• A Linux environment for closer-to-production command-line behavior.
• A benchmark dataset with task metadata, intermediate states, and ground-truth patches or outputs.
• A results database or structured log format for storing run traces and score components.
• Statistical analysis tools for comparing model rankings and confidence intervals.

Software & Tools

• Docker: Creates the same sandboxed environment for every run so your benchmark stays reproducible.
• Python: Runs scoring scripts, parses logs, and computes reliability metrics.
• Pandas: Organizes run data so you can compare models, tasks, and score components.
• Jupyter Notebook: Lets you inspect results, make plots, and document analysis in one place.
• ImageJ: Not used here, so skip it unless you also analyze screenshots from a related interface study.

Experiment Steps

1. Define the failure mode you want to measure, such as losing state, picking the wrong tool, or recovering poorly after an error.
2. Choose a small task set that matches your benchmark goal and gives you enough variety to test long-horizon behavior.
3. Design the metric so it scores more than final success, then decide how you will weight intermediate actions and errors.
4. Build a reproducible run pipeline in Docker so every model sees the same environment and task setup.
5. Plan a validation test that compares your new score against a simpler baseline, such as task completion rate.
6. Set up an analysis plan that checks repeatability, ranking stability, and whether the metric reacts to expected changes in agent behavior.

Common Pitfalls

• Scoring only the final output, which hides agents that took bad paths but happened to finish.
• Using tasks that are too easy, which creates ceiling effects and makes every model look similar.
• Letting the Docker environment drift between runs, which breaks reproducibility and weakens your claims.
• Designing a metric that rewards verbose action logs instead of real task progress.
• Comparing models on different task subsets, which makes the leaderboard look fair when it is not.

What Makes This Competitive

A stronger version of this project goes past a simple pass or fail score. You would build a metric that captures reliability across multiple failure modes, then prove it matches human intuition better than a basic completion score. You could also test whether your benchmark stays stable across repeated runs and different task mixes. If your analysis shows clear ranking differences and a reason for them, your project starts to look like real systems research.

Project Variations

• Measure reliability on file-editing tasks instead of full software fixes to see whether the metric generalizes to smaller CLI workflows.
• Compare open-source models and commercial APIs on the same sandbox tasks to study how tool-use reliability changes across model families.
• Test whether adding error-recovery sub-scores improves ranking stability more than using only final task success.

Learn More

• SWE-bench: Search the project paper and repository to study how software-engineering tasks are turned into benchmark problems.
• Docker Documentation: Read the official docs for sandboxing, containers, and reproducible environments on the Docker website.
• Python Documentation: Use the official Python docs for scripting, log parsing, and simple data analysis.
• Pandas Documentation: Find examples for tabular analysis, filtering, grouping, and summary statistics in the Pandas user guide.
• PubMed: Search for review articles on LLM evaluation, tool use, and benchmark reliability, especially papers about agent benchmarks.
• arXiv: Search for recent preprints on agentic tool use, software engineering benchmarks, and long-horizon evaluation.