LLM Reward Shaping for Robot Manipulation

ISEF Category: Robotics and Intelligent Machines

Subcategory: Machine Learning  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

A robot can spend thousands of tries learning a task if the reward signal is weak. That is like trying to get better at basketball while only hearing, “good job” after a perfect shot. Your project asks if an LLM can act like a coach and give better feedback. If it works, the robot learns faster with less guesswork.

What Is It?

This project studies reward shaping, which means changing the feedback a robot gets during training. In reinforcement learning, a robot learns by trying actions and receiving rewards. If the reward is sparse, the robot may only get points at the very end of a task. That can make learning slow. Dense rewards give smaller signals along the way, which can guide the robot more smoothly.

An LLM, or large language model, can read a task description and judge whether the robot is getting closer to success. Think of it like a coach who reads the rulebook and gives step-by-step feedback. Instead of hand-writing every reward rule, you ask the model to score progress from the task text and the robot state. Then you compare that approach with standard hand-engineered rewards, which are human-written formulas based on task progress.

Why This Is a Good Topic

This topic works well because you can test a clear question, does LLM-based feedback help a robot learn faster than a human-designed reward? You also get a real-world connection to robot training, where reward design can take a lot of time. The math and coding are advanced, but the idea is easy to explain, measure, and compare. You can learn about reinforcement learning, evaluation metrics, and fair benchmarking.

Research Questions

• How does LLM-generated dense reward compare with sparse reward in sample efficiency for one MetaWorld task?
• What is the effect of prompt style on the quality of LLM-generated rewards?
• Does task complexity change how well LLM-based reward shaping helps learning?
• To what extent does LLM reward shaping match hand-engineered reward performance across multiple manipulation tasks?
• Which task descriptions lead to the most stable reward signals from the LLM?
• How does adding state variables to the prompt change the robot's final success rate?

Basic Materials

• Computer with a modern GPU or access to a university machine-learning server.
• Python environment with PyTorch and MetaWorld installed.
• Access to a large language model API or a local open-weight model, if allowed by your lab.
• Code editor such as VS Code.
• Git for version control.
• Spreadsheet software or Python notebooks for tracking experiments.
• Plotting library such as Matplotlib or Seaborn.
• Basic statistics tools for comparing learning curves.

Advanced Materials

• University GPU workstation or cluster access.
• MetaWorld simulation environment and compatible reinforcement learning framework.
• Large language model API access or a locally hosted model with inference tools.
• Experiment tracking software such as Weights & Biases or MLflow.
• Docker or Conda for reproducible environments.
• Python libraries for reinforcement learning, logging, and evaluation.
• Statistical analysis tools for confidence intervals, significance tests, and learning curve smoothing.
• Optional ablation setup for prompt variants, state features, and reward aggregation rules.

Software & Tools

• Python: Runs the reinforcement learning code, reward pipeline, and analysis scripts.
• PyTorch: Trains the agent and supports custom reward integration.
• MetaWorld: Provides the simulated manipulation tasks used for benchmarking.
• Jupyter Notebook: Helps you inspect learning curves, logs, and prompt outputs.
• Matplotlib: Plots reward trends, success rates, and sample efficiency comparisons.

Experiment Steps

1. Define one manipulation task and the exact success metric you will use.
2. Choose the baseline reward design, then decide how your LLM-based critic will receive task and state information.
3. Plan a fair comparison setup so both reward methods train the same agent under the same budget.
4. Build a logging scheme that records success rate, reward shape, and learning speed across runs.
5. Design ablation tests that change only one prompt or state input at a time.
6. Predefine the statistics you will use to compare sample efficiency and final performance.

Common Pitfalls

• Giving the LLM too little state information, which makes its reward guesses noisy and inconsistent.
• Comparing methods with different training budgets, which makes the sample-efficiency result unfair.
• Using prompt text that leaks the answer, which can make the reward look better than it really is.
• Testing only one random seed, which hides how unstable reinforcement learning can be.
• Measuring only final success rate and ignoring how quickly the agent learned during training.

What Makes This Competitive

A stronger project would test more than one task and more than one prompt design. You could measure learning speed, final success, and reward stability, then use confidence intervals or seed-based statistics. A very competitive version would also compare the LLM critic against several human-written reward variants, not just one baseline. That gives you a sharper claim about when language-based reward shaping helps and when it does not.

Project Variations

• Test the same reward-shaping idea on a different simulated manipulation suite, such as a pick-and-place task with a different object set.
• Compare an LLM critic with a small learned reward model to see whether text reasoning or data-driven scoring works better.
• Use the same framework to study prompt sensitivity, where you change the wording of the task description and track reward consistency.

Learn More

• MetaWorld paper and code: Search for the MetaWorld benchmark paper on arXiv and the public GitHub repository for task definitions and evaluation details.
• OpenAI Spinning Up: Free reinforcement learning background and policy-gradient explanations, available from OpenAI's educational site.
• MIT OpenCourseWare 6.036: Search MIT OpenCourseWare for machine learning lectures that cover reinforcement learning basics.
• arXiv: Search for recent papers on LLMs, reward shaping, and robotic manipulation to find current methods and baselines.
• PubMed: Search for review articles on human feedback, agent learning, and decision support if you want a broader view of feedback systems.