LLM Code Watermarking That Survives Editing

ISEF Category: Systems Software

Subcategory: Cybersecurity  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

AI-generated code can look clean and still hide a watermark. The hard part is keeping that watermark alive after someone renames variables or reformats the file. That makes this project a real test of how durable code marks can be in the wild. You get to think like both the creator and the attacker.

What Is It?

A watermark is a hidden signal that proves where something came from. In this project, the thing you mark is code written by a large language model, or LLM. The goal is to leave a trace that still survives common edits, like changing variable names or reformatting indentation.

Think of it like signing a drawing in a way that still survives a photocopy. A weak signature disappears if someone trims the page. A strong one stays readable even after copies, scans, and small edits. Your job is to test how much code can change before the watermark breaks.

This topic sits at the intersection of cybersecurity, software engineering, and machine learning. You are not just asking, “Can we hide a signal?” You are asking, “Can the signal survive realistic attacks and transformations?” That makes the project both technical and practical.

Why This Is a Good Topic

This is a strong science fair topic because you can measure real outcomes, compare methods, and define an attacker model with clear rules. You can test how well a watermark survives transformations such as identifier renaming, code formatting changes, and small refactors. That connects to real problems in AI accountability, authorship, and misuse detection. A student can learn evaluation design, benchmarking, and statistical comparison without building a full model from scratch.

Research Questions

• How does identifier renaming affect watermark detection accuracy in LLM-generated code?
• What is the effect of code reformatting on watermark survival across multiple programming tasks?
• Does the watermark remain detectable after common semantic-preserving edits such as variable renaming, function extraction, or whitespace normalization?
• To what extent does the attacker model change false positive and false negative rates?
• Which transformation breaks the watermark first, formatting changes or renaming changes?
• How does watermark strength trade off against normal code quality measures such as pass rate on unit tests?

Basic Materials

• Laptop or desktop computer.
• Python installed locally.
• Code editor such as VS Code or PyCharm Community Edition.
• Access to HumanEval or a similar public code benchmark.
• Publicly available LLM-generated code samples or your own generated sample set.
• Spreadsheet software for tracking test results.
• Git for version control and reproducible runs.

Advanced Materials

• University workstation or server access.
• Python research environment with libraries for code parsing and evaluation.
• AST tooling for the target language, such as tree-sitter or a language-native parser.
• Benchmark datasets such as HumanEval and The Stack, accessed under their published terms.
• Unit test harness for code execution and scoring.
• Statistical analysis environment such as R or Python with SciPy and pandas.
• Secure sandbox for running generated code safely.

Software & Tools

• Python: Runs analysis scripts, transformation tests, and result summaries.
• pandas: Organizes benchmark outputs, attack types, and detection scores.
• matplotlib: Plots watermark survival curves and error rates.
• tree-sitter: Parses code into syntax trees so you can test structure-aware edits.
• Git: Tracks code versions and keeps experiments reproducible.

Experiment Steps

1. Define the attacker model you will test, including which edits count as allowed transformations.
2. Choose one watermarking method and one comparison baseline so you can measure improvement fairly.
3. Build a test set of code samples and plan how you will generate or collect them from public benchmarks.
4. Design a transformation pipeline that applies renaming, reformatting, and other code edits in a controlled way.
5. Decide the metrics you will report, such as detection accuracy, false positives, false negatives, and task success rate.
6. Plan a comparison table that separates watermark survival from overall code correctness.

Common Pitfalls

• Testing only on one programming language, which hides whether the watermark generalizes.
• Mixing up code that still works with code whose watermark still survives, which are two different outcomes.
• Using a weak attacker model, which makes the watermark look better than it really is.
• Forgetting to check for false positives on clean, unwatermarked code.
• Changing multiple transformations at once, which makes it hard to tell what actually broke the watermark.

What Makes This Competitive

A strong version of this project would compare several attacker models and several watermark designs, then report where each one fails. You could add syntax-aware and semantics-aware edits, not just cosmetic ones. Strong analysis would separate watermark detectability from code correctness, since a watermark that survives but breaks the program is not useful. Careful statistics, clear baselines, and a well-defined threat model would make the work feel much more serious.

Project Variations

• Test whether the watermark survives Python code transformations more often than JavaScript transformations.
• Compare syntax-tree-based renaming attacks with simple text-based renaming attacks.
• Measure how watermark detection changes when you apply transformations to HumanEval tasks versus The Stack samples.

Learn More

• PubMed: Search for review articles on watermarking, authorship detection, and AI model security if you want background on evaluation style.
• arXiv: Search for recent preprints on LLM code watermarking and robustness to edits.
• MIT OpenCourseWare: Look for introductory materials on algorithms, software testing, and computational thinking.
• The Stack documentation: Read the public dataset description and access notes for code benchmark context.
• HumanEval paper: Find the original benchmark paper and related citations through arXiv or the publisher page.
• NIH PubMed Central: Search for papers on program analysis, software robustness, and adversarial evaluation methods.