Federated Gesture Recognition for Smart Home Control

ISEF Category: Robotics and Intelligent Machines

Subcategory: Machine Learning  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

Your smartwatch can know a gesture, but your raw motion data should not have to leave the device. That privacy tradeoff sits at the center of federated learning. You can test whether several small wearables can learn the same gesture model without sharing their raw IMU data. Then you can compare the accuracy and network cost against a centralized system.

What Is It?

This project studies federated learning, a training method where devices learn together without sending raw data to a central server. Instead of uploading every accelerometer and gyroscope reading, each wearable trains on its own motion data and sends model updates. Think of it like a study group where everyone shares notes, not their whole notebook.

The gesture-recognition part uses IMU data, which means motion signals from sensors such as accelerometers and gyroscopes. The goal is to tell one gesture from another, like a swipe, tap, or raise-to-wake motion. In your project, the wearables would act like tiny smart-home controllers. A successful model could map a hand gesture to a home action while keeping the user’s motion traces on-device.

Why This Is a Good Topic

This is a strong science fair topic because you can measure clear numbers, like classification accuracy, training rounds, bandwidth used, and model size. You can also compare two systems, federated and centralized, so your conclusion has a built-in benchmark. The topic connects to real privacy problems in wearables, smart homes, and health sensors. You can learn machine learning design, experimental control, and performance tradeoffs that matter in real products.

Research Questions

• How does federated training with 3 wearables compare to centralized training for gesture classification accuracy?
• What is the effect of the number of local training rounds on federated model accuracy?
• Does adding a fourth wearable improve generalization across users more than training on 3 wearables?
• To what extent does model compression reduce communication cost before accuracy drops sharply?
• Which gesture classes are hardest for a federated model to separate from IMU data?
• How does non-identical data across wearables affect convergence speed and final accuracy?

Basic Materials

• ESP32-S3 development boards with IMU sensors, or ESP32-S3 boards plus separate IMU modules.
• Three or more wearable straps, clips, or small enclosures for mounting boards.
• USB cables for each device.
• Laptop with Python installed.
• MicroSD cards or local storage, if the boards need data logging.
• Smartphone or tablet, if you want a simple control interface.
• Digital stopwatch or logging app for tracking test sessions.
• Notebook or spreadsheet for labeling gestures and sessions.

Advanced Materials

• ESP32-S3 development boards with onboard IMU, or comparable low-power microcontrollers.
• Lab-grade motion capture system, if available, for validating gesture labels.
• Wireless access point or network monitor for measuring packet traffic.
• External battery packs with voltage logging.
• Edge AI accelerator board, if you want a stronger local baseline.
• Reference computer for centralized training experiments.
• Calibration jig or fixed mount for repeatable motion testing.
• Secure data storage system for managing participant traces.

Software & Tools

• Python: Trains models, processes IMU data, and compares federated and centralized results.
• Jupyter Notebook: Helps you clean data, test features, and graph accuracy trends.
• TensorFlow or PyTorch: Builds the gesture classifier and supports custom training loops.
• scikit-learn: Provides metrics, splitting tools, and baseline models for comparison.
• Wireshark: Measures network traffic and helps you estimate communication cost.

Experiment Steps

1. Define the gesture set, the user group, and the smart-home actions you want to classify.
2. Decide how you will split data across wearables so the federated setup matches real privacy limits.
3. Build a centralized baseline first, because you need a benchmark before you test federated learning.
4. Plan your model update rule, communication schedule, and stopping rule for the federated system.
5. Set up metrics for accuracy, per-class confusion, bandwidth, and training rounds so you can compare systems fairly.
6. Design control tests that separate user variation, sensor noise, and network effects from the learning method itself.

Common Pitfalls

• Mixing gesture labels across sessions, which makes the model learn the wrong motion classes.
• Comparing federated and centralized models with different train-test splits, which ruins the benchmark.
• Letting each wearable collect data in slightly different body positions, which adds sensor bias instead of useful variation.
• Reporting only overall accuracy, which hides weak performance on rare or similar gestures.
• Ignoring communication overhead, which makes a heavy federated method look better than it really is.

What Makes This Competitive

A strong version of this project goes beyond a simple accuracy check. You can compare multiple federated settings, different gesture sets, and a careful centralized baseline under the same data split. You can also test whether privacy-preserving training hurts some users more than others, which adds a real fairness angle. Strong analysis, clear controls, and clean communication-cost measurements can make the project feel much more like published engineering research.

Project Variations

• Use wrist, ankle, and pocket-mounted wearables to test how body placement changes federated gesture recognition.
• Compare raw-model sharing with compressed or quantized updates to study the accuracy and bandwidth tradeoff.
• Test personalization by fine-tuning the federated model on one user after global training finishes.

Learn More

• MIT OpenCourseWare, Introduction to Machine Learning: Search MIT OpenCourseWare for machine learning lectures and assignments that explain training, validation, and overfitting.
• TensorFlow Federated Tutorials: Free examples from Google that show how to build federated learning experiments.
• PyTorch Tutorials: Free guides for building and testing neural networks on sensor data.
• NIH PubMed: Search for review articles on wearable sensor gesture recognition and privacy-preserving learning.
• IEEE Xplore abstracts: Search for recent papers on federated learning for wearable activity or gesture recognition, then read available abstracts and open-access papers.
• NIST Smart Spaces resources: Search NIST for work on IoT, edge devices, and security tradeoffs in connected systems.