Sentiment Analysis of German Sign Language Fairy Tales
Fabrizio Nunnari, Siddhant Jain, Patrick Gebhard
TLDR
This paper introduces a dataset and an XGBoost model for sentiment analysis of German Sign Language (DGS) fairy tales, revealing key facial and body features.
Key contributions
- Created a dataset for sentiment analysis of German fairy tale texts using LLMs, with 0.781 Krippendorff's alpha.
- Extracted face and body motion features from corresponding DGS video segments using MediaPipe.
- Developed an explainable XGBoost model to predict sentiment from DGS video features, achieving 0.631 accuracy.
- Found that face (eyebrows, mouth) and body (hips, elbows, shoulders) are equally important for DGS sentiment.
Why it matters
This paper pioneers sentiment analysis for German Sign Language, offering a new dataset and an explainable model. Its findings highlight the critical role of both facial and body movements in conveying sentiment in sign language, advancing our understanding of DGS communication.
Original Abstract
We present a dataset and a model for sentiment analysis of German sign language (DGS) fairy tales. First, we perform sentiment analysis for three levels of valence (negative, neutral, positive) on German fairy tales text segments using four large language models (LLMs) and majority voting, reaching an inter-annotator agreement of 0.781 Krippendorff's alpha. Second, we extract face and body motion features from each corresponding DGS video segment using MediaPipe. Finally, we train an explainable model (based on XGBoost) to predict negative, neutral or positive sentiment from video features. Results show an average balanced accuracy of 0.631. A thorough analysis of the most important features reveal that, in addition to eyebrows and mouth motion on the face, also the motion of hips, elbows, and shoulders considerably contribute in the discrimination of the conveyed sentiment, indicating an equal importance of face and body for sentiment communication in sign language.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.