Unifying Diverse Logs in Big Data Systems for Seamless Analysis and Action With OpenSe...
summary
Pipeline ingests Databricks Spark listener logs from S3 via AWS Lambda into an OpenSearch vector store, converts log lines to embeddings, and stores relationships in a graph DB for lineage so RAG queries return job‑ and task‑scoped context. LLMs plus a Python tooling function compute practical fixes (example: derive 50 shuffle partitions from 6.4GB shuffle), synthetic datasets validate changes cheaply, and an agent layer runs parallel experiments — cutting problem‑finding from ~5–8 hours to ~5–15 minutes.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 2:32Big data systems overview
- 6:31Debugging Spark jobs
- 11:30Spark log analysis
- 15:26LLMs for Spark logs
- 18:19RAG architecture
- 21:54Graph databases for RAG
- 25:25Solution docs in RAG
- 27:14Dynamic solution recommendations
- 30:23Synthetic dataset validation
- 32:10Agentic AI validation
- 33:37Automated troubleshooting demo
- 35:23Incremental feedback UX
- 37:09AI memory and domain knowledge
- 38:14Guardrails in agentic AI
- 40:22Scalable AI practices
- 41:05Q&A
related talks
01h 35mDeep Dive into the Big Data Landscape
29mWide Event Analytics
53mNoLambda: A new architecture combining streaming, ad hoc, machine...
44mBest practices for analytics and generative AI on AWS(ANT302)
30mData @Scale Seattle- Sundaram Narayanan
51mQuery Understanding With Large Language Models: Techniques and...
40mBuilding Intelligent Search Systems With Agentic RAG and OpenSearch
58mAI-Enhanced Big Data: Integrating Private LLMs and Vector Databases