Data Intelligence Platform on AWS
summary
A practical data-catalog assistant is deployed that uses a retrieval-augmented pipeline: metadata from S3/Delta Lake feeds a vector index (tested with OpenSearch vector store), retrieval augments prompts, and LLMs (Bedrock models like Titan or external ChatGPT/Claude) generate answers and SQL. Integration uses Redshift UDF via Lambda and a Slack bot. Result: over 500 internal users, a nine‑person team, 100+ databases and ~1,000 raw tables, fewer support tickets, some SQL generation errors, and iterative metadata fixes.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 4:03Data platform goals
- 10:18Data catalog challenges
- 16:07AI assistant for discovery
- 24:01Simplifying BI with Amazon Q
- 32:14User independence impact
- 35:46Conclusion
related talks
24mBuilding Your Intelligent Data Agent
45mManaging Data & Analytics Projects Effectively
24mMetadata driven framework for data discovery and data quality
32mPromoting a data driven culture in a world of microservices
43mFuture-proofing your applications with AWS databases
28mBuilding database queries with Amazon Bedrock
04mInterview
14mMulticloud Analytics Made Simple