From Multimodal Signals to Adaptive XR Experiences for De-escalation Training
Birgit Nierula, Karam Tomotaki-Dawoud, Daniel Johannes Meyer, Iryna Ignatieva, Mina Mottahedin + 2 more
TLDR
A new multimodal system uses real-time signals like speech, gestures, and biometrics to create adaptive XR de-escalation training experiences.
Key contributions
- Presents a multimodal system integrating five real-time signal streams for adaptive XR training.
- Analyzes verbal, gestural, affective, mental state, and physiological cues simultaneously.
- Links low-level signals to de-escalation constructs via an instructor-informed interpretation layer.
- Demonstrates feasibility in XR de-escalation training, highlighting multi-view fusion benefits.
Why it matters
This paper is crucial for advancing human-AI interaction in complex XR training environments. It offers a robust framework for real-time analysis of user states, enabling highly adaptive and effective de-escalation simulations. The insights on multimodal fusion address key challenges in XR sensing.
Original Abstract
We present the early-stage design and implementation of a multimodal, real-time communication analysis system intended as a foundational interaction layer for adaptive VR training. The system integrates five parallel processing streams: (1) verbal and prosodic speech analysis, (2) skeletal gesture recognition from multi-view RGB cameras, (3) multimodal affective analysis combining lower-face video with upper-face facial EMG, (4) EEG-based mental state decoding, and (5) physiological arousal estimation from skin conductance, heart activity, and proxemic behavior. All signals are synchronized via Lab Streaming Layer to enable temporally aligned, continuous assessments of users' conscious and unconscious communication cues. Building on concepts from social semiotics and symbolic interactionism, we introduce an interpretation layer that links low-level signal representations to interactional constructs such as escalation and de-escalation. This layer is informed by domain knowledge from police instructors and lay participants, grounding system responses in realistic conflict scenarios. We demonstrate the feasibility and limitations of automated cue extraction in an XR-based de-escalation training project for law enforcement, reporting preliminary results for gesture recognition, emotion recognition under HMD occlusion, verbal assessment, mental state decoding, and physiological arousal. Our findings highlight the value of multi-view sensing and multimodal fusion for overcoming occlusion and viewpoint challenges, while underscoring that fusion and feedback must be treated as design problems rather than purely technical ones. The work contributes design resources and empirical insights for shaping human-AI-powered XR training in complex interpersonal settings.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.