Sim-to-Real Tactile Slip Detection

ISEF Category: Robotics and Intelligent Machines

Subcategory: Machine Learning  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

A robot hand can miss a slip by a fraction of a second, and that tiny delay can ruin a grasp. Your project asks whether a cheap camera and a silicone pad can catch slip before the object falls. The twist is that you train the model on fake data first, then see if it still works on the real world. That gap between simulation and reality is where the science gets interesting.

What Is It?

This project studies tactile slip detection. Slip happens when an object starts moving against a gripper before it fully escapes. A human feels that shift fast, but a robot needs a sensor and a model to notice it. Your setup uses a camera pressed against a soft silicone pad, so tiny surface changes become visible as texture motion in the image.

Think of the pad like a fingernail pressed into gelatin. When an object pushes or slides, the surface pattern changes shape. Optical flow measures that motion between frames. A classifier then learns patterns that suggest slip onset, which means the first moment the object starts to slide instead of stay fixed.

Why This Is a Good Topic

This is a strong science fair topic because you can vary one thing at a time, such as the training data, the object surface, or the motion pattern, and measure how prediction accuracy changes. It connects to real problems in robotics, prosthetics, warehouse automation, and safe gripping. You can learn computer vision, supervised learning, feature design, and model validation without needing a full industrial robot. The sim-to-real part gives you a clear question that goes beyond a simple demo.

Research Questions

• How does training on synthetic contact patches affect slip-detection accuracy on real silicone pad images?
• What is the effect of different optical-flow features on the early detection of slip onset?
• Does adding simulated lighting variation improve transfer from PyBullet images to camera images?
• To what extent does the classifier generalize across different object surface textures?
• Which model type, such as logistic regression, random forest, or CNN, best predicts slip onset from tactile image sequences?
• How does the amount of synthetic training data change precision and recall on real grasp sequences?

Basic Materials

• Raspberry Pi Camera Module or similar small camera.
• Raspberry Pi or laptop for image capture and model testing.
• Printed silicone pad or a soft translucent gel pad.
• Stable frame or clamp to hold the camera against the pad.
• Objects with different surface textures, such as plastic, rubber, fabric, and wood.
• Good lighting setup with fixed LEDs.
• Tape, clips, and mounting materials.
• Spreadsheet software for tracking trials and results.

Advanced Materials

• High-resolution industrial or machine-vision camera.
• Custom molded silicone taxel pad with embedded markers.
• 3D-printed mounting rig for repeatable contact geometry.
• Force sensor or load cell for labeling slip onset.
• Motion stage or robotic arm for controlled contact trials.
• Calibration target for camera distortion correction.
• Dedicated GPU workstation for model training.
• MATLAB or Python environment with computer vision libraries.

Software & Tools

• Python: Runs data cleaning, feature extraction, model training, and evaluation.
• OpenCV: Computes optical flow and handles image preprocessing.
• PyBullet: Generates synthetic contact scenes for training data.
• scikit-learn: Trains and compares classic classifiers on extracted features.
• ImageJ: Helps inspect pad images, align frames, and measure surface changes.

Experiment Steps

1. Define the slip event you will predict, and decide how you will label the first moment of motion.
2. Design a repeatable camera-pad setup so your images stay aligned across trials.
3. Build a synthetic dataset that matches your real contact scenes as closely as possible.
4. Choose a small set of optical-flow features, then decide how you will convert them into classifier inputs.
5. Plan a fair test split that keeps real trials separate from the simulated data used for training.
6. Select evaluation metrics that reflect early warning quality, not just raw accuracy.

Common Pitfalls

• Training only on perfect synthetic images, which makes the model fail when real lighting and blur appear.
• Labeling slip after the object has already moved far enough to make the onset time unclear.
• Changing camera distance or angle between trials, which changes optical-flow patterns more than the slip itself.
• Using one object texture for training and expecting the classifier to work on every other surface.
• Reporting accuracy alone, which hides missed slips when the classes are unbalanced.

What Makes This Competitive

A stronger version of this project compares more than one sim-to-real strategy and explains why one transfer method works better. You can test whether domain randomization, feature selection, or model choice matters most. You can also use stricter metrics like precision, recall, F1 score, and detection latency. A competitive project makes a clear claim about what improves transfer, then backs it up with clean validation.

Project Variations

• Train the classifier on one set of synthetic lighting conditions, then test whether lighting randomization improves real-world transfer.
• Replace optical-flow features with raw image patches or edge maps, and compare which input type predicts slip onset best.
• Test whether the model transfers across multiple pad materials, such as silicone, gel, and foam, to see how sensor texture changes generalization.

Learn More

• PyBullet Documentation: Learn how to build physics simulations and synthetic contact scenes, then find the docs through the official project site.
• OpenCV Documentation: Read about optical flow, image preprocessing, and camera calibration in the official docs and tutorials.
• scikit-learn User Guide: Compare classifiers, metrics, and validation methods in the free official documentation.
• MIT OpenCourseWare, Introduction to Machine Learning: Review core ideas on supervised learning, features, and model evaluation.
• PubMed: Search review articles on tactile sensing, slip detection, and robotic grasp stability.