Real-Time Control of a Virtual Orchestra by Recognition of Conducting Gestures
Mert Mermerci, Emile Pascoe, Fredrik Edström, Hedvig Kjellström
TLDR
A museum installation allows visitors to conduct a virtual orchestra in real-time using vision-based gesture recognition and an LSTM network.
Key contributions
- Developed a museum installation for conducting a virtual orchestra in a 180° dome theater.
- Utilizes a vision-based skeleton tracker to capture visitor gestures for real-time playback control.
- Employs a hierarchical LSTM network, trained on multiple conductors, for gesture recognition.
- System evaluated via timing accuracy, user studies on realism/usability, and a field study with visitors.
Why it matters
This paper introduces an innovative interactive museum experience, making classical music conducting accessible to the public. It demonstrates a robust real-time gesture control system, validated with extensive user studies. Its practical application in a public setting highlights its potential for immersive educational entertainment.
Original Abstract
We present a museum installation in a 180° dome theater, which gives the museum visitor the experience of conducting a symphony orchestra. We have pre-recorded a short music piece performed by a professional orchestra. This recording is played back in the dome with the visitor standing in the conductor's position. The visitor's gestures are captured with a vision-based skeleton tracker, steering the recording playback pace via a gesture recognition module that translates the gestures into a time control signal. This is sent to a playback module that plays the recording in the dome at the corresponding speed. The gesture recognition module is based on a hierarchical LSTM network, trained with recorded sequences of multiple conductors with different level of expertise conducting the same recording. The system is evaluated with a quantitative study of the estimated timing accuracy, a user study evaluating the musical realism and usability of the real-time control, and a field study to evaluate the performance of the entire system with real museum visitors.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.