Dynamic Training in AR for Musical Practice
Augmented reality application for musical practice with real-time feedback and interactive training.
Project
Augmented reality application for musical practice with real-time feedback and interactive training.
01
Learning context
The project investigates how an augmented mirror can support musical imagery, pitch awareness and deliberate practice through immediate audiovisual feedback.
02
Theremin prototype
The first prototype mapped the position of both hands to gain and pitch, creating a virtual theremin. Visual bars and particles showed the playable frequency space and the current hand positions.
03
Design iteration
Pose-estimation latency made rhythmic performance and accurate pitch selection uncomfortable. The interaction was redesigned so that movement and sound were no longer directly coupled.
04
Pitch-training system
The revised module listens to a singer or instrument, estimates the emitted frequency and visualizes it on the mirror. Exercises display target notes over time and provide feedback when the performed pitch matches.
05
Architecture
The application runs on GOSAI with a P5.js interface. A microphone driver and frequency-estimation module publish audio features through Redis and Socket.IO to the browser display.