Smartphone Tongue AI for Health Screening

ISEF Category: Biomedical Engineering

Subcategory: Biomedical Sensors and Imaging  ·  Difficulty: Advanced  ·  Setup: Home Setup  ·  Time: Full Year

The Hook

Your tongue can carry clues about hydration, anemia, and vitamin status. A phone camera can catch some of those clues if you control the lighting and color. That makes this project part image analysis, part medical sensing, and part AI. You are not diagnosing anyone, you are testing whether a simple camera system can spot patterns worth a closer look.

What Is It?

This project asks whether tongue photos can help screen for health-related signals such as dehydration, anemia, or B12 deficiency. The idea is simple. A healthy tongue may look different from one affected by low blood volume, low iron, or nutrient issues. But raw phone photos change a lot because of light, skin tone, camera model, and angle. So you add a color-calibration card to make images more comparable.

A convolutional neural network, or CNN, is an AI model that learns visual patterns from images. Think of it like a very fast pattern detector. It can learn which color, texture, and shape features line up with each label. Grad-CAM then helps you see which parts of the image the model used most. That matters because a strong project does not just predict. It also checks whether the model is paying attention to the tongue surface, edges, and coating instead of random background noise.

Why This Is a Good Topic

This is a strong science fair topic because you can test a real engineering problem, not just build a demo. You can measure whether color calibration improves model accuracy, whether self-collected images help more than a public dataset alone, and whether explainability maps look medically sensible. The topic connects to low-cost health screening, which matters in places with limited access to clinics. You can also learn imaging, data labeling, model training, and validation.

Research Questions

• How does adding a color-calibration card affect CNN accuracy for tongue-image classification?
• What is the effect of training on self-collected images plus the public TCM tongue dataset versus the public dataset alone?
• Does Grad-CAM highlight tongue regions that match known visual cues for dehydration, anemia, or B12 deficiency?
• To what extent does camera type change model performance after color correction?
• Which image preprocessing method, such as white balance correction or histogram normalization, gives the best classification results?
• How does class imbalance affect sensitivity for the rarest health label?

Basic Materials

• Smartphone camera with manual or pro mode.
• Color-calibration card or printed color reference chart with known colors.
• Tripod or phone stand.
• Consistent light source, such as a ring light or desk lamp with a diffuser.
• Plain background, such as gray or black poster board.
• Free image annotation tool, such as Label Studio or CVAT.
• Laptop or desktop computer with enough memory for image processing.
• Spreadsheet software for tracking image labels and metadata.
• Consent form and data-collection plan approved by a school or mentor review process.
• Mirror and clean water for standardizing tongue photo setup.

Advanced Materials

• Smartphone with RAW capture support.
• Color target with certified patches, such as X-Rite ColorChecker if available through school or lab access.
• Reference camera and multiple phone models for transfer testing.
• Computer with GPU access for CNN training.
• Python environment with OpenCV, TensorFlow, or PyTorch.
• ImageJ or Fiji for image inspection and color measurements.
• Annotation platform for labeling tongue regions and quality control.
• Secure storage for de-identified image metadata.
• Optional clinical partner access for supervised, consented collection of labeled images.
• Statistical software for ROC, confusion matrix, and calibration analysis.

Software & Tools

• Python: Scripts image preprocessing, model training, and performance analysis.
• OpenCV: Corrects color, crops images, and checks lighting consistency.
• TensorFlow: Trains a CNN for image classification.
• PyTorch: Trains and tests image models, including explainability workflows.
• ImageJ: Measures color and texture features in tongue regions.
• Grad-CAM tools: Generates heatmaps that show which image areas influenced the model.

Experiment Steps

1. Define the exact labels you will predict and decide whether you are building a screening model or a feature study first.
2. Plan a photo protocol that keeps tongue position, lighting, and camera distance as consistent as possible.
3. Build a calibration workflow that turns raw phone photos into comparable images across sessions and devices.
4. Organize your dataset into training, validation, and test groups so the same person does not leak across splits.
5. Choose a baseline model and compare it against a version that uses color correction and any added preprocessing.
6. Design an explainability check with Grad-CAM and compare the highlighted regions with your expected biological cues.

Common Pitfalls

• Collecting images under changing light, which makes color differences look like health signals when they are really camera noise.
• Mixing photos from the same person across training and test sets, which inflates accuracy and hides overfitting.
• Using labels that are too vague, which makes the model learn muddled classes instead of clear targets.
• Ignoring class imbalance, which can make the model look good overall while failing on the rarest condition.
• Trusting Grad-CAM maps without sanity checks, which can hide the fact that the model is reading the background or lips instead of the tongue.

What Makes This Competitive

A competitive version of this project would go beyond a simple classifier. You would test whether calibration truly improves generalization across phones, skin tones, and lighting setups. You would also compare multiple models, report class-specific metrics, and check whether the explainability maps match known visual patterns. Strong entries usually ask a harder question than, “Can AI classify images?” They ask, “When does it work, why does it work, and when does it fail?”

Project Variations

• Focus only on dehydration screening and compare tongue moisture features across different lighting setups.
• Compare CNN performance with and without the public TCM tongue dataset to test domain transfer.
• Replace classification with regression to predict a continuous tongue-color or coating score from calibrated images.

Learn More

• PubMed: Search for review articles on tongue diagnosis, anemia screening, dehydration, and oral imaging in peer-reviewed journals.
• NIH National Library of Medicine: Use PubMed and related medical resources to find studies on oral biomarkers and image-based screening.
• NOAA Color and Light resources: Review basic color calibration and imaging concepts that help with consistent photo capture.
• NASA Image Analysis resources: Look for free materials on image preprocessing, feature extraction, and visualization workflows.
• MIT OpenCourseWare, Introduction to Machine Learning: Use free lecture notes and assignments to understand CNN basics and evaluation.