Gesture2Music
Low-latency real-time framework for continuous gesture-driven music generation
Gesture2Music is a real-time gesture-driven music generation system that maps continuous human gestures into musical outputs. The project uses computer vision and temporal sequence modeling to support low-latency interaction between body movements and generated music.
Role: Collaborating researcher
Key contributions:
- Built gesture-recognition workflows using MediaPipe-based face and hand landmarks.
- Developed temporal modeling pipelines for recognizing gesture patterns.
- Supported real-time inference for gesture-driven interaction.
- Contributed to continuous gesture-to-music generation and evaluation.
Related publication:
Rathinaraja Jeyaraj, Barathi Subramanian, Kapilya Gangadharan, Anand Paul, “Gesture2Music: A Low-Latency Real-Time Framework for Continuous Gesture-Driven Music Generation,” First International Workshop on Interactive Physical AI, IEEE/CVF CVPR 2026.