Reduce cost and capacity planning burden with Glue Auto Scaling
summary
Glue Auto Scaling is generally available and automatically adjusts Glue worker count per stage to cut cost and remove capacity tuning. Jobs start with a lightweight driver that does query planning and optimization (charged only for driver during planning), then requests workers only when needed; common examples include JDBC ingestion that begins with few connections and later scales for transforms, and small-file handling using file grouping and pushdown predicate. Enablement is a single checkbox or API flag.
AI-written summary · about
chapters (14)
- 0:00Intro
- 1:35AWS Glue: Key capabilities
- 8:24Interface for every persona
- 9:38AWS Glue: Recap of key innovations
- 12:08Scalable execution model
- 13:05Business events impact capacity and cost
- 14:19Predicting compute demands is difficult
- 16:21How to enable Auto Scaling
- 16:47Job containing multiple stages
- 20:36Large number of small files
- 21:15AWS Glue optimization: Pushdown predicate
- 22:07Glue optimization: File grouping
- 22:51Comparison: With and without Auto Scaling
- 25:22Glue Auto Scaling: Putting it all together
related talks
44mOptimize Costs with Serverless Data Integration Using AWS Glue
38mUnleashing the serverless data pipelines
13mHow to automate legacy ETL migration to AWS Glue
16mAutomated Data Analytics
20mAutomate the modernization of legacy ETL to AWS Glue using LeapLogic
28mML + Big Data: Spark on Amazon SageMaker with Amazon Glue
21mWhy should you bother about cluster autoscaling- Karpenter
27mInnovation through self-service data products