LLM Compiler Bug Minimizer Project

ISEF Category: Systems Software

Subcategory: Languages and Operating Systems  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

A tiny test case can save hours of debugging. Compiler teams use shrinking tools to cut a crash down to the smallest possible example, then fix the real bug faster. Your project asks a simple question with big stakes, can an LLM help find a smaller crash case than classic minimizers?

What Is It?

This project is about compiler bug minimization. A compiler takes code you write and turns it into machine code. When it crashes, the original test case is often too large to inspect by hand. A minimizer tries to trim that test case until only the bug-triggering core stays.

Think of it like peeling an onion. Each layer you remove should not change the crash. Tools like C-Reduce do this by repeatedly editing the input and checking whether the bug still happens. Your idea adds an LLM, which can suggest smarter edits or guide the search toward smaller, cleaner test cases.

The key outcome is not just whether the crash still happens. You care about size, success rate, and time. That gives you a clear way to compare the new method against a classic baseline.

Why This Is a Good Topic

This is a strong science fair topic because you can measure it with real compiler crashes and objective numbers. You can test whether one method produces smaller reproducing examples, needs fewer trials, or keeps the bug alive more often. The work connects to software reliability, debugging, and automated program analysis. You can also learn how to design fair benchmarks, compare algorithms, and reason about search strategies.

Research Questions

• How does an LLM-guided minimizer compare with C-Reduce on final test case size?
• What is the effect of different prompt styles on the minimizer’s success rate?
• Does adding bug report text to the prompt improve reduction quality?
• To what extent does the approach work better on optimization-pipeline crashes than on parser crashes?
• Which edit strategy produces the smallest reproducing test case per compiler bug category?
• How does the method’s runtime compare with C-Reduce across the same bug corpus?

Basic Materials

• Laptop or desktop computer with enough storage for compiler builds and bug datasets.
• Internet access for downloading source code, issue reports, and benchmark cases.
• Local installation of GCC and or Clang build tools.
• Python 3 with a virtual environment manager.
• Git for version control and dataset tracking.
• Text editor or IDE such as VS Code.
• Spreadsheet software for logging results.
• Basic shell access on Linux or macOS, or WSL on Windows.

Advanced Materials

• Access to a machine with multiple CPU cores for repeated compiler runs.
• Docker or a reproducible build environment for isolating compiler versions.
• LLVM source tree and historical bug corpus from the LLVM bug tracker.
• C-Reduce or a comparable reducer for baseline testing.
• Scriptable harness for compiling, running, and scoring candidate test cases.
• Database or structured log store for tracking each reduction attempt.
• Optional access to a GPU if you test local LLM inference or embedding-based ranking.
• Profiling tools such as perf or time for measuring search cost.

Software & Tools

• Python: Automates test generation, reduction loops, and result analysis.
• Git: Tracks experiment code, prompts, and benchmark cases.
• GCC: Provides one compiler target for crash reproduction and reduction tests.
• Clang and LLVM: Provide the optimization pipeline and bug corpus for evaluation.
• ImageJ: Not needed here, so skip it and use text and log analysis instead.

Experiment Steps

1. Define one compiler crash family and one success metric before you write any code.
2. Build a baseline reduction pipeline so you can compare your method against C-Reduce under the same conditions.
3. Choose the LLM input format, including what bug context, code context, and edit history you will provide.
4. Design a fair scoring rule for smaller size, preserved crash behavior, and search cost.
5. Plan a benchmark set from historical LLVM bugs, then split it so you can test on unseen cases.
6. Set up result logs that let you compare methods bug by bug, not just by average performance.

Common Pitfalls

• Testing only one compiler version, which makes your minimizer look better or worse for the wrong reason.
• Comparing against C-Reduce without matching the same timeout, seed, or starting test case.
• Measuring only final file size and ignoring whether the reduced case still reproduces the crash.
• Letting the LLM see the answer case during prompt design, which leaks benchmark information into the search.
• Mixing parser bugs, optimization bugs, and backend bugs in one pool, which hides where the method actually works.

What Makes This Competitive

A stronger project would test more than one bug family and report where the method wins, where it fails, and why. You could add ablation studies that remove bug report text, edit history, or prompt structure to see which part helps most. A careful statistical comparison, plus a clean reproducible harness, would make the work much more convincing. If you also explain failure modes, your project looks like real research, not just a tool demo.

Project Variations

• Compare LLM-guided reduction against C-Reduce on bugs from only one compiler, such as Clang.
• Test whether bug report text helps more than raw source code when the model proposes edits.
• Measure whether the method works better on syntax-valid crashes than on crashes that require specific optimization passes.

Learn More

• LLVM Bug Tracker: Search historical compiler bugs and reduction cases in the LLVM issue tracker.
• C-Reduce paper: Read the original reducer paper in a peer-reviewed systems venue, then search for it through your school library or Google Scholar.
• GCC documentation: Learn how compiler optimization flags and passes behave from the official GCC manuals.
• LLVM documentation: Read about Clang and LLVM optimization pipelines in the official project docs.
• PubMed: Search for review articles on automated program repair and program synthesis to see related methods, even though this topic is software-focused.