Animal ID App Bias Audit

ISEF Category: Animal Sciences

Subcategory: Other  ·  Difficulty: Intermediate  ·  Setup: Home Setup  ·  Time: 1 to 2 Months

The Hook

An app can name a bird in seconds, but that does not mean it knows every bird equally well. If the training data skews toward one region, the app can miss animals from elsewhere. That gap is a bias problem, and you can measure it with real data.

What Is It?

Think of these apps like a friend who has seen tons of sparrows and only a few tropical birds. The friend gets the common ones right, then guesses badly when the animal is unfamiliar. Bias here does not always mean bad intent, it means the model has seen some species, places, or photo styles much more than others.

Seek and Merlin are animal-ID tools that turn a photo or sound into a best guess. Your job is to ask where those guesses stay strong, where they break down, and whether the mistakes cluster by region or by rare taxa, which are species groups with too few examples. Then you can sketch a retraining plan that gives underrepresented taxa more weight in a new model or test set.

Why This Is a Good Topic

You can test this with public observations, app outputs, and a spreadsheet. It connects to biodiversity, machine learning bias, and citizen science, so the project has real-world value. You learn data cleaning, confusion matrices, and how to judge a model by more than one accuracy score.

Research Questions

• How does app accuracy change across animal observations from different world regions?
• What is the effect of taxon rarity on top-1 identification accuracy?
• Does accuracy differ between common and underrepresented taxa within the same habitat type?
• To what extent do image quality and metadata completeness change app confidence scores?
• Which species groups produce the highest confusion rates in each region?
• How does a class-balanced retraining set change performance on rare taxa in a prototype model?

Basic Materials

• Smartphone or tablet with Seek installed.
• Smartphone or tablet with Merlin Bird ID installed.
• Laptop with spreadsheet software such as Google Sheets or Excel.
• Internet access for public observation records from iNaturalist and GBIF.
• Curated set of public animal photos or bird sounds with region labels.
• Notebook or digital form for logging app guesses, target species, and confidence.

Advanced Materials

• Laptop with Python, pandas, scikit-learn, and GeoPandas.
• Jupyter Notebook for analysis and plotting.
• Downloaded observation exports from iNaturalist and GBIF.
• QGIS for mapping regional sample coverage.
• Reference taxonomy list from Catalogue of Life or ITIS.
• External drive or cloud storage for image and audio archives.

Software & Tools

• Python: Cleans data, scores app outputs, and compares accuracy across regions.
• Jupyter Notebook: Keeps code, charts, and notes in one place.
• pandas: Sorts records, joins metadata, and builds comparison tables.
• scikit-learn: Trains a simple baseline classifier for a retraining test.
• QGIS: Maps where the sample set is thin or overrepresented.

Experiment Steps

1. Define the regions, taxa, and app versions you will compare.
2. Build a labeled test set from public records with balanced coverage across regions.
3. Decide the metrics you will report, such as top-1 accuracy, confusion rate, and calibration.
4. Plan a blind scoring method so you record each app guess the same way every time.
5. Design a retraining strategy using class balancing, sample weighting, or targeted rare-taxon additions.
6. Set up a comparison that checks whether rare taxa improve without pushing common taxa down.

Common Pitfalls

• Mixing species from regions with very different camera angles, which makes region effects look like app bias.
• Using far more common species than rare ones, which hides the size of the error on underrepresented taxa.
• Judging the app only by its first guess, which misses useful information in its confidence ranking.
• Letting saved images vary in resolution or cropping, which turns photo quality into a hidden variable.
• Comparing taxa names without checking taxonomy updates, which creates false mismatches between the app and your reference list.

What Makes This Competitive

A strong version of this project does more than list accuracy. It compares error patterns by region, taxa rarity, and input quality, then tests whether a balanced retraining set actually fixes the weakest classes. If you add calibration, confusion matrices, and a clear holdout set from regions the model never saw, your analysis starts to look like real model evaluation. That turns a simple app check into a serious fairness study.

Project Variations

• Compare bird-only results from Merlin with insect and plant results from Seek to see whether one app is more region-sensitive than the other.
• Audit how image quality and background clutter change identification accuracy for the same species set across continents.
• Test a small open-source classifier retrained on underrepresented taxa and compare it with the app baseline on a holdout set.

Learn More

• GBIF: Search species occurrence records and download regional data for coverage checks.
• iNaturalist Help Center: Learn how observations are labeled and filtered, then find it on the iNaturalist website.
• PubMed: Search review articles on algorithmic bias, species recognition, and biodiversity informatics.
• USGS BISON: Explore North American species distribution data and find it through the USGS site.
• MIT OpenCourseWare: Watch free machine learning lectures to understand model evaluation and class imbalance.
• Catalogue of Life: Check accepted species names and synonyms for taxonomy cleanup.