Non-Linear Consumption of Videos Using a Sequence of Personalized Multimodal Fragments
summary
NoVoExp generates a sequence of image+sentence multimodal fragments by extracting keyframes, timestamped transcripts, contiguous clusters, and selecting representative text and image conditioned on viewer preference, with preference labels learned from the Pitts Ads dataset. The system is evaluated on YouTube videos across three categories (entertainment, lectures, tutorials) and short/medium/long durations, and human AMT studies show fragment descriptions match full-video descriptions and rate high for relevance, diversity, coherence and jolly and fun-loving preference alignment, striking a practical balance across metrics and working in practice.
AI-written summary · about
chapters (7)
- 0:00Start
- 3:56Extract Out Key Visual and Auditory Information from the Video
- 4:21Contiguous Multimodal Clusters
- 4:55Selecting a Representative Multimodal Fragment for each Cluster
- 5:47Define User Preferences
- 6:31Evaluation
- 7:11Automated Evaluation
related talks
56mApplications of Multimodal Learning in Media Search Engines
33mArchitecting Scalable Multi-Modal Video Search
40mThe Marvelous Magic of Multimodal AI
23mML-Powered Search & Recommendation 101
30mVideo Quality Assessment of User Generated Contents
18mCoVA: Exploiting Compressed-Domain Analysis to Accelerate Video Analytics
23mAVA: Towards Agentic Video Analytics with Vision Language Models
30mMLDS 2019: Anubhav Shrivastava of Voot at Machine Learning Developers Summit 2019