The Doctor Will See You Now: Machine Learning for Telehealth
summary
Built a browser telehealth app called dialer video that reaches over 80% of US physicians and ran hundreds of thousands of clinician calls and millions of call minutes, and implements virtual backgrounds using TensorFlow.js plus MediaPipe/selfie segmentation (uploader also cites BodyPix). Real time constraint targets ~30 FPS so segmentation runs in ~33 ms, mask smoothing uses a bilateral filter via glfx.js, compositing uses Canvas globalCompositeOperation, demo ties into Twilio video, works in Chromium with OffscreenCanvas and is production deployed.
AI-written summary · about
chapters (13)
- 0:00Start
- 6:48Chroma-key Compositing
- 8:40Image segmentation
- 12:14Computer vision
- 12:34Compositing a virtual background
- 12:56Compositing in 4 steps
- 20:52Create the video processor
- 21:38Process the video frame
- 22:58Paint the mask
- 23:20Apply a bilateral filter
- 23:57Step 2.1: Crop the bilateral filter
- 24:23Composite the foreground
- 24:44Composite the background
related talks
24mState of client side machine learning
17mTransforming access to healthcare through Web AI
51mRealTime Exoskeleton-Mapping and Body Segmentation on the Browser with Tensorflow.js
29mSuperpowers for next gen web apps: Machine learning
01h 56mCase Study + Deep Dive: Telemedicine Support Agents with LangGraph/MCP
15mMachine learning for next gen web apps with TensorFlow.js
58mPushing Deepfakes to the Limit-Fake video calls with AI
24mWeb Apps of the Future With Web AI