Offline Pill Recognition App for Medicine Tracking

ISEF Category: Biomedical and Health Sciences

Subcategory: Other  ·  Difficulty: Intermediate  ·  Setup: Home Setup  ·  Time: 1 to 2 Months

The Hook

One wrong pill guess can turn a routine dose into a real problem. Your app works like a spotter for tablets, matching a photo to a labeled database and logging whether the dose was taken. The hard part is not just recognizing pills, but recognizing them when the room is dark, the counter is cluttered, or the phone angle shifts.

What Is It?

Computer vision means software that reads images instead of text. In this project, the app looks at a pill photo and predicts the label by comparing shape, color, imprint, and size patterns. The offline part means the phone keeps working without sending images to the internet.

The dose-adherence tracker adds a memory layer. After the app identifies the pill, it can record a check-off, a reminder, or a missed dose. That matters for older adults, because a simple interface and no internet requirement can make the tool easier to use at home.

Why This Is a Good Topic

This is a strong science fair topic because you can test it with clear inputs and clear outputs. Lighting, background clutter, angle, and phone distance all change image quality, so you can measure how each factor changes accuracy. The project also connects to a real problem, because medication mix-ups and missed doses can affect daily safety. You can learn image classification, app design, and evaluation without needing a professional lab.

Research Questions

• How does lighting level affect top-1 pill ID accuracy?
• What is the effect of background clutter on recognition confidence?
• Does adding imprint-only crops improve classification accuracy?
• To what extent does phone distance change offline inference speed?
• Which camera angle produces the highest identification accuracy?
• How does reminder timing affect missed-dose logging in a simulated user test?

Basic Materials

• Android phone or iPhone with a working camera.
• Laptop with Python 3.10 or newer.
• Public pill-image dataset from NLM Pillbox or RxImage.
• Spreadsheet software or Google Sheets.
• Desk lamp or another steady light source.
• Plain white, black, and patterned backgrounds.
• Ruler or tape measure for keeping camera distance consistent.
• Optional phone stand or tripod.

Advanced Materials

• GPU workstation or cloud notebook for model training.
• TensorFlow or PyTorch environment.
• Android test phones with different camera sensors.
• Color calibration card for image consistency checks.
• Light box or controlled photo booth.
• High-resolution camera rig for repeatable captures.
• Annotated pill image set with imprint and shape labels.
• Mobile benchmarking tool for on-device latency testing.

Software & Tools

• Python: Trains models, cleans data, and scores accuracy.
• OpenCV: Preprocesses images and tests how lighting and blur affect recognition.
• TensorFlow Lite: Converts the model for offline phone use.
• Android Studio: Builds and runs the mobile app on test devices.
• Google Colab: Trains early prototypes without local GPU hardware.

Experiment Steps

1. Define the narrow pill set you will support, such as one shape or one brand family, so your labels stay clean.
2. Split your data by pill type and by source image before training, so nearly identical photos do not leak into the test set.
3. Choose one baseline model and one improved version, then decide what changes between them, such as cropping, normalization, or quantization.
4. Build an evaluation plan that records top-1 accuracy, false-match rate, and on-device latency under different lighting and background conditions.
5. Design the offline tracker flow, including how the app stores the dose log, reminder state, and failed recognition results without internet access.
6. Set up a comparison chart that shows which pill classes break first, because look-alike tablets often reveal the real limits of the system.

Common Pitfalls

• Training and testing on nearly identical pill photos, which inflates accuracy and hides weak generalization.
• Using only clean, centered images, which makes the app fail when the pill sits on a countertop or hand.
• Ignoring look-alike pills with similar color and imprint, which increases dangerous false matches.
• Reporting one overall score instead of per-pill confusion results, which hides the hardest cases.
• Forgetting offline constraints during development, which leaves you with a model that works only when the internet is on.

What Makes This Competitive

A strong version compares models across controlled lighting, background, and blur, not just one clean test set. You can also report top-1 and top-3 accuracy, confusion patterns for look-alike pills, and latency on different phones. A more competitive entry would test whether imprint crops, color normalization, or quantization helps each pill subgroup, then use confidence intervals or a paired significance test to show the differences are real. The best angle is safety, not just accuracy, because false matches matter more than average performance.

Project Variations

• Compare full-frame pill photos against tight imprint crops to see which input helps the model more.
• Focus on one medication class, such as tablets, capsules, or gelcaps, and test how far lighting changes can push accuracy.
• Add a reminder-use study that measures whether the app logs missed doses more reliably than a simple checklist interface.

Learn More

• NLM Pillbox: Search the National Library of Medicine's pill image and imprint database for labeled pill examples.
• RxImage: Use this NIH pill identification resource for reference images and imprint data.
• PubMed: Search for review articles on medication adherence, pill identification, and mobile health.
• OpenCV Documentation: Read the free docs for image preprocessing, camera handling, and blur or lighting checks.
• TensorFlow Lite Guide: Find the official guide for exporting a model to run offline on a phone.