On-Device Food Portion Estimation App

ISEF Category: Systems Software

Subcategory: Mobile Apps  ·  Difficulty: Intermediate  ·  Setup: Home Setup  ·  Time: 1 to 2 Months

The Hook

A dinner photo can hide a lot. A burger can look small or huge depending on angle, plate size, and lighting. Your phone can still estimate portion size if you give it one real-world reference, like a credit card. That turns a casual snapshot into a measurement problem.

What Is It?

This project asks whether a phone camera can estimate food portion size without cloud help. The app first finds the food in the image. That step is called segmentation, which means separating the food from the background. Then it uses a known object, like a credit card or fingernail, as a scale reference so the app can estimate size and volume.

Think of it like tracing a shadow and then using a ruler to guess the size of the object that made it. The photo gives shape, the reference object gives scale, and the model turns pixels into an estimate. You can then compare your app's nutrition estimate with labels from Nutritionix, which gives you a real benchmark for accuracy.

Why This Is a Good Topic

This topic works well because you can test it with ordinary photos, clear measurements, and real benchmarks. You do not need a wet lab, and you can still ask a real engineering question about accuracy, speed, and privacy. It connects to food logging, nutrition tracking, and mobile health tools. You can learn computer vision, basic model evaluation, and how design choices affect real-world performance.

Research Questions

• How does the choice of scale reference, such as a credit card or fingernail, affect portion estimate accuracy?
• What is the effect of plate color contrast on food segmentation quality?
• Does on-device processing change estimate speed compared with a cloud-based workflow?
• To what extent does food shape, such as flat, piled, or irregular, change volume estimation error?
• Which segmentation method gives the best agreement with Nutritionix labels for mixed meals?
• How does lighting variation affect the stability of the estimated calorie count?

Basic Materials

• Smartphone with a rear camera and local storage,
• Credit card or card-sized reference object,
• Fingernail or another small body-based reference for comparison,
• Standard plates, bowls, and trays,
• Variety of foods with clearly different shapes and colors,
• Measuring cup or kitchen scale for ground-truth checks,
• Spreadsheet software for logging image features and error values.

Advanced Materials

• Smartphone with on-device machine learning support,
• Laptop for model testing and annotation,
• Annotation tool for image masks,
• Open-source segmentation model or mobile vision framework,
• Reference objects with known dimensions,
• Kitchen scale for ground-truth mass checks,
• Nutrition database access for label comparison,
• Image dataset storage with consistent naming and version control.

Software & Tools

• ImageJ: Measures object area and helps you compare segmentation masks across photos.
• Python: Lets you calculate error metrics, run comparisons, and organize your analysis.
• Google Sheets: Tracks image conditions, reference type, and prediction error in one table.
• Label Studio: Supports manual image annotation if you need to build your own mask set.
• Nutritionix: Provides nutrition labels you can use as the benchmark for your app output.

Experiment Steps

1. Define the exact output your app will predict, such as portion area, estimated volume, or calories.
2. Choose one scale reference and one fallback reference, then decide how you will compare them fairly.
3. Plan a photo set that varies food type, plate type, lighting, and camera angle while keeping other factors stable.
4. Build a measurement pipeline that separates food pixels from background pixels and converts that shape into a numeric estimate.
5. Set up controls that let you compare app output with a ground-truth source, such as Nutritionix labels or weighed portions.
6. Predefine the error metrics you will report, such as percent error, mean absolute error, or agreement rate.

Common Pitfalls

• Using only one type of food, which makes the app look accurate on simple cases and weak on real meals.
• Choosing a scale reference that is partly hidden or warped by perspective, which throws off the size estimate.
• Changing lighting between photos, which shifts segmentation quality and makes results hard to compare.
• Comparing app output to Nutritionix labels without matching serving definitions, which creates a false error signal.
• Testing only one camera angle, which hides how badly the model fails when the phone is held higher or lower.

What Makes This Competitive

A strong version of this project does more than show that an app can guess portions. It tests several reference objects, several food types, and several error metrics. It also separates segmentation error from volume estimation error, so you can say where the system fails. If you compare on-device performance against a cloud workflow, and back your claims with clean statistics, the project starts to look much stronger.

Project Variations

• Test whether a credit card outperforms a fingernail as the scale reference across different meal types.
• Compare single-item foods with mixed plates to see how segmentation error changes with scene complexity.
• Measure whether a fully offline model stays accurate after compression for faster on-device processing.

Learn More

• OpenCV documentation: Learn basic image segmentation and contour measurement, and find it through the OpenCV official docs.
• MIT OpenCourseWare, Computer Vision: Review core ideas in image processing and object detection, and find it on MIT OpenCourseWare.
• PubMed: Search review articles on food image analysis, dietary assessment, and portion estimation.
• NIH 3D Print Exchange: Explore measurement and modeling ideas that help with object size estimation, and find it through NIH resources.
• Nutritionix database: Use public nutrition label data as a comparison source, and find it by searching the Nutritionix site.
• IEEE Xplore: Search for papers on mobile food recognition, portion estimation, and on-device vision systems.