Bowed String Articulation Classifier for Sheet Music

ISEF Category: Technology Enhances the Arts

Subcategory: Music and Image Manipulation  ·  Difficulty: Advanced  ·  Setup: School Lab  ·  Time: Full Year

The Hook

A violin note can sound smooth, sharp, bouncy, or scratchy, and the bow is the reason why. Your phone hears all of that as one messy wave. Your project is to teach a model to tell those playing styles apart in real time, then use that signal to light up sheet music.

What Is It?

This project uses sound recognition to identify how a string player is using the bow. Legato means the notes connect smoothly. Staccato means the notes are short and separated. Spiccato means the bow bounces. Col legno means the player taps or strikes the string with the wood of the bow. These are called articulations, and they change the feel of the music just like punctuation changes a sentence.

You would record the instrument with a contact mic, which picks up vibration through the instrument instead of only room sound. Then you turn each recording into a mel-spectrogram, which is a picture of sound that shows how energy changes across frequencies over time. A 1D-CNN, or one-dimensional convolutional neural network, learns patterns in those spectrogram features and classifies the articulation. The final step is to move the model onto an ESP32-S3, a small microcontroller that can run the classifier during practice and light up the next required marking in the score.

Why This Is a Good Topic

This is a strong science fair topic because you can test a real machine learning pipeline from data collection to live deployment. You can compare different mics, feature sets, window sizes, and model sizes, so the project has clear variables and measurable outcomes. It connects to music education, assistive practice tools, and low-power edge AI. You can also learn audio processing, model evaluation, and embedded deployment, which are all useful research skills.

Research Questions

• How does microphone type affect articulation classification accuracy for bowed string recordings?
• What is the effect of using mel-spectrograms versus raw audio features on model performance?
• Does adding more articulation classes reduce classification accuracy more for some bowing styles than for others?
• To what extent does instrument type, violin versus cello, change the model's ability to generalize?
• Which window size gives the best tradeoff between live response time and classification accuracy?
• How does training on one player versus multiple players affect real-world performance?

Basic Materials

• Hobby violin or cello.
• Contact microphone with 3.5 mm output or USB audio interface.
• Laptop or desktop computer with microphone input.
• Tripod or phone stand for fixed recording position.
• Quiet practice space.
• Sheet music with marked articulation examples.
• Notebook or spreadsheet for labels and observations.

Advanced Materials

• Contact microphone array or high-quality piezo contact mic.
• Audio interface with low-noise preamp.
• ESP32-S3 development board.
• Small OLED, LED strip, or e-ink display for score highlighting.
• MicroSD card module for local logging.
• Calibrated reference speaker or bowing setup for repeatable trials.
• Python environment for model training and export.

Software & Tools

• Python: Processes audio, trains the classifier, and evaluates results.
• librosa: Extracts mel-spectrograms and other audio features from recordings.
• TensorFlow Lite: Converts the trained model for edge use on the ESP32-S3.
• Audacity: Checks recordings for noise, clipping, and bad labels.
• ImageJ: Can help inspect spectrogram images if you export them as visuals.

Experiment Steps

1. Define the articulation labels and decide how you will keep each sample cleanly tied to one playing style.
2. Plan your recording setup so the mic position, instrument, and room stay as consistent as possible across sessions.
3. Design a feature pipeline and decide whether you will train on raw audio, spectrograms, or both.
4. Build a comparison plan for model size, player variation, and instrument variation so you can test generalization.
5. Choose an edge deployment target and confirm the model fits the speed and memory limits of the ESP32-S3.
6. Plan a live feedback display that turns predictions into a clear practice cue without distracting the player.

Common Pitfalls

• Using recordings from only one bow angle, which makes the model memorize one setup instead of the articulation.
• Letting room noise or string squeaks dominate the signal, which confuses the classifier.
• Mixing up labels for short notes and bounced bow strokes, which blurs the difference between staccato and spiccato.
• Training and testing on clips from the same take, which inflates accuracy and hides weak generalization.
• Ignoring class imbalance, which can make the model look good while failing on rare articulations like col legno.

What Makes This Competitive

A stronger version of this project would test whether the model works across different players, instruments, and recording setups instead of only one sample set. You could compare several feature pipelines and measure not just accuracy, but also confusion between similar articulations. If you add edge deployment, latency, and memory limits to the analysis, the project starts to look like real product research. A careful error study will matter more than a flashy demo.

Project Variations

• Use flute or clarinet tonguing patterns instead of bowed articulations, and compare whether the same audio pipeline still works.
• Test the classifier on live ensemble rehearsal recordings rather than isolated practice clips.
• Replace the LED score cue with a tablet overlay that highlights articulation mistakes in real time.

Learn More

• MIT OpenCourseWare, 6.036 Introduction to Machine Learning: Search the course page for neural networks, classification, and evaluation basics.
• TensorFlow Lite Micro documentation: Read the official guides for running small models on microcontrollers.
• librosa documentation: Use the examples to learn audio loading, feature extraction, and mel-spectrograms.
• NIH PubMed: Search for review articles on music performance, sensor-based feedback, and machine learning in audio analysis.
• IEEE Xplore or ACM Digital Library: Search for papers on sound classification, embedded AI, and instrument articulation recognition.