Tabletop Theory of Mind Robot

ISEF Category: Robotics and Intelligent Machines

Subcategory: Cognitive Systems  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

A robot that helps before you ask sounds smart, but smart robots can also be very wrong. In a block game, one mistaken handoff can ruin the whole trial. Your project asks a simple but deep question, can a small language model guess what a person wants from context alone? That puts you right at the edge of robotics, language, and human teamwork.

What Is It?

This project studies theory of mind in robots. Theory of mind means guessing what another person knows, wants, or believes, even when that person does not say it out loud. In your case, the robot watches a cooperative block-stacking game, infers the human partner's hidden goal, and chooses a block to hand over.

Think of it like a helpful friend who notices you are building a tower, not a wall, and passes you the next piece you need. A scripted robot follows fixed rules. A model-based robot tries to read the situation and predict the next move. You can test whether the model really does better on false-belief trials, where the human partner has a mistaken or hidden goal that the robot must infer from context.

Why This Is a Good Topic

This is a strong science fair topic because you can measure it clearly. You can count correct handoffs, wrong handoffs, response time, and performance across different game setups. The project connects to real problems in assistive robots, warehouse robots that work near people, and social robots that need to cooperate instead of just react. You can also learn prompt design, experiment design, logging, and statistical comparison.

Research Questions

• How does a small LLM robot's false-belief task accuracy change when the human partner's goal is hidden versus stated outright?
• What is the effect of adding turn-by-turn context on the robot's ability to hand the correct block?
• Does a scripted baseline make more incorrect handoffs than an LLM-based system in cooperative block-stacking?
• To what extent does the robot's prediction accuracy depend on the number of visible blocks on the table?
• Which prompt style produces the fewest wrong block handoffs in ambiguous game rounds?
• How does the robot's response time change when it must infer a hidden goal instead of follow a direct request?
• To what extent do prior successful rounds influence the robot's later predictions in the same session?

Basic Materials

• Laptop or mini PC that can run Ollama locally.
• Webcam or overhead camera for recording the game table.
• Small tabletop robot or robotic arm with a simple gripper.
• Assorted colored building blocks or wooden cubes.
• Printed game boards or marked play area to keep trials consistent.
• Notebook or spreadsheet for logging each trial.
• Tripod or fixed camera mount for repeatable recordings.
• Timer or timestamped video recording for response time checks.
• Human participant consent form and trial instructions.

Advanced Materials

• Mobile robot or tabletop manipulator with API access.
• USB camera or depth camera for object detection and tracking.
• Local machine capable of running Phi-3 or Llama-3.2-1B through Ollama.
• Python environment with robotics, vision, and data analysis libraries.
• Object detection model or color segmentation pipeline for block identification.
• Microphone if you want spoken instructions instead of text prompts.
• Motion capture or pose tracking system if available.
• Database or structured log format for storing trial history.
• Statistical analysis software or notebook for mixed-effects analysis.

Software & Tools

• Ollama: Runs a small language model locally so you can test inference without cloud costs.
• Python: Handles trial logging, prompt formatting, and data analysis.
• ImageJ: Measures block positions and helps you score camera-based trials frame by frame.
• Pandas: Organizes trial data, prediction labels, and accuracy metrics.
• Jupyter Notebook: Keeps your analysis, plots, and notes in one place.

Experiment Steps

1. Define the hidden-goal task so every trial has a clear correct block and a clear wrong block.
2. Build a baseline policy that follows simple rules, then separate it from the LLM version so you can compare them fairly.
3. Design the prompt and context window so the robot sees only the information a real partner would have seen.
4. Plan your scoring rules before data collection, including what counts as correct, late, ambiguous, or unsafe behavior.
5. Set up repeated trials with matched layouts, matched participants, and matched block sets so you can isolate the effect of inference.
6. Choose your analysis plan, then compare accuracy, response time, and error types across conditions.

Common Pitfalls

• Giving the model too much hidden information, which lets it guess the answer without real inference.
• Changing the block layout between trials, which makes accuracy comparisons meaningless.
• Scoring a vague handoff as correct, which inflates performance on ambiguous rounds.
• Letting camera angle or table position drift, which changes what the robot can detect from one session to the next.
• Comparing the LLM system only to a weak scripted baseline, which makes the result look better than it really is.

What Makes This Competitive

A competitive version of this project would test more than raw accuracy. You could compare multiple prompt styles, multiple hidden-goal types, and multiple participant behaviors, then analyze where the robot fails. Strong projects also report confusion cases, not just averages. If you can show that one design improves false-belief reasoning without slowing the robot too much, your work starts to look like real human-robot interaction research.

Project Variations

• Test the same hidden-goal setup with colored cups or cards instead of blocks.
• Compare a text-only robot prompt with a vision-assisted robot that also reads the table layout.
• Analyze whether the robot does better with one human partner across many rounds or with several new partners.

Learn More

• MIT OpenCourseWare, Introduction to Robotics: Search for robotics lectures that cover sensing, planning, and human-robot interaction.
• PubMed: Search for review articles on theory of mind, social robots, and human-robot collaboration.
• IEEE Xplore: Search for papers on cognitive robotics and cooperative task planning, then read abstracts and methods.
• ArXiv: Search for recent work on small language models for embodied agents and robot planning.
• NASA Tech Briefs: Search for articles on autonomous systems and human-machine teaming concepts.