Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership...
summary
Introduces a neuroscience-inspired topological regularization applied to BLIP, PaLI-Gemma 2 and ViT-GPT2 fine-tuned on COCO, CC3M and NoCaps, measuring membership inference via caption similarity (MPNet, ROUGE-2) and attack ROC-AUC. Reports neuro (τ=2) cuts BLIP+COCO mean ROC-AUC from ~94% to ~70% and neuro++ (τ=3) to ~63%, while MPNet/ROUGE-2 stay comparable; effectiveness varies by architecture, dataset and regularization strength.
AI-written summary · about
chapters (13)
- 0:00Neuro-inspired VLMs and privacy
- 1:12Research questions and methods
- 3:41Membership inference attacks
- 6:09Experimental setup and metrics
- 7:13Topological regularization framework
- 9:18Mathematical formulation and implementation
- 10:01Results on COCO dataset
- 11:27Caption generation analysis
- 12:51Privacy-utility trade-off
- 13:57Data granularity impact
- 14:37Limitations and implications
- 15:20Key takeaways
- 15:41Q&A
related talks
46mAttention to Detail: Fine-Grained Vision-Language Alignment
24mMulti-Modal AI Agents Explained | Open-Vocabulary Alignment
01h 08mMastering Visual AI with Vision-Language Models & Advanced Evaluation Techniques
16mBLens: Contrastive Captioning of Binary Functions using Ensemble Embedding
44mAI Intuition: Exploring Language Model Latent Space
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence
10mLearning Network-Based Multi-Modal Mobile User Interface Embeddings
40mGraph neural networks for biomedical insights