Bringing Interactivity to Videos
summary
Built a timed metadata pipeline that carries interactive objects from creation through delivery and playback, with live events packed into emsg MP4 segments and VOD stored in a separate metadata track. An inject API accepts RTMP streams from third‑party encoders; heavy ML/AR processing moves into custom FFmpeg filters near pixel transcoding to avoid per‑frame seconds of extra latency. Playback runs a monitor → decoder → interpreter that synchronizes stickers, polls, 3D ads and questionnaires to video and audio.
AI-written summary · about
chapters (9)
- 0:00Interactive video framework
- 0:48Lean-back vs lean-forward
- 1:31Interactivity benefits
- 2:36Interactivity pipeline
- 4:49Challenges in pipelines
- 7:16Advanced capabilities & latency
- 9:02Playback rendering framework
- 10:53Example applications
- 13:00Interactive polls & engagement
related talks
44mDesigning for Interactive Games
23mCreating Videos Programmatically in React
08mSimulating livestreams for testing
08m10x your video experience: Why Mux is your Strapi instance's best friend
09mTransform live video streams with code and a REPL!!
16mJavaScript can be the missing spork in ffmpeg's swiss army knife
19mImproving DX and adoption of Membrane Framework
34mLive Q&A with Speakers