Technical Workshop: Sarcastically Speaking: Unlocking Multi-modal Sentiment Analysis With NLP and Facial Expressions
summary
Workshop builds a full pipeline from MP4 -> frames -> CSV landmarks and MP4 -> WAV -> acoustic CSV, uses the Mustard dataset with heavy cleaning, dlib 68-point landmarks and a nose-tip anchoring normalization, extracts RMS/HNR/jitter/shimmer and word-level timestamps, trains LSTM facial and LSTM+CNN audio model and fuses scores. Results: facial F1 ≈78.3%, audio ≈76%, multimodal speaker-dependent ≈80% (original paper 72%), speaker-independent ≈70% (original 62%); works well but fails on deadpan or off-camera cases.
AI-written summary · about
chapters (11)
- 0:00Multimodal sarcasm detection intro
- 2:50Sarcasm challenges for text models
- 6:02MUSTARD dataset start
- 11:06Cleaning video data
- 14:57Facial landmarks with dlib
- 21:21Facial expression model
- 25:19Audio sarcasm clues
- 28:51Audio feature pipeline
- 35:45Combining multimodal models
- 38:43Future improvements
- 42:00Workshop materials & Q&A
related talks
15mRecognizing Emotional Expression in Game Streams
39mSentiment Analysis, its techniques and applications
25mLooking at Sound
39mThe art and science of making computers understand sentiment and emotion
37mA Practical Guide to Robust Multi modal Machine Learning in Education
31mOff-the-shelf HuggingFace models for audio deepfake detection
23mPrediction of music pairwise preferences from facial expressions
20mCapturing, Representing, and Interacting with Laughter