Best practices for analytics and generative AI on AWS(ANT302)
summary
Pipeline ingests sensitive telemetry from 18 million endpoints and moves ~5 TB per day into regional S3 landing zones using Amazon MSK and VPC endpoints, writing Avro for appendability and later converting to Parquet for query efficiency. Processing uses AWS Glue for cataloging, Amazon EMR (ephemeral clusters, instance fleets, spot) for large Spark jobs, Step Functions for orchestration, Athena and Lake Formation workspaces for governed access, DynamoDB for the versioned knowledge base, and Bedrock (Claude 3.5) via Lambda/LangChain; batching ~20 items and 5x sampling improves LLM classification confidence on an index built from clustered embeddings of app descriptions and a 100 billion events single-run scale example.
AI-written summary · about
chapters (18)
- 0:00Introduction and speakers
- 1:07Generative AI applications overview
- 3:36In-context learning patterns
- 6:47AWS services for generative AI
- 8:14Real-time data pipeline setup
- 9:37Optimizing data on S3
- 11:02Data processing with AWS Glue
- 14:18Amazon EMR best practices
- 17:49NexThink generative AI workflow
- 18:10NexThink company overview
- 21:23Data Lake architecture at NexThink
- 23:55Pre-processing pipelines and orchestration
- 26:44Data Lake use cases and access control
- 29:12Generative AI for product features
- 31:19Categorizing applications with LLMs
- 36:59Integrating Amazon Bedrock
- 38:25Iterating on generative AI solutions
- 41:54Key takeaways and conclusion
related talks
28mTurn Data Into Insights
22mTurn Data Into Insights
21mSmarter, faster Data analytics with generative AI and Machine Learning
42mAgentic AI with AWS Streaming
21mOptimize and manage data on Amazon S3
25mA simple serverless data pipeline
27mHow AWS helps Variant employ a driver-first data-driven strategy
16mAutomated Data Analytics