Harnessing the Power of Dual Query Engines.
summary
Workday built a dual-query platform combining Trino/Presto for interactive SQL and Spark/Livy for ETL and ML, fronted by a custom data access service exposing a unified REST API and CLI. Engineering reports 3–10× faster query latency on Trino vs Spark, enforces governance (resource groups, max memory/time, partition filters), and runs daily JVM restarts to avoid severe slowdowns. The system runs on S3/Glue/EMR/Hive, migrated thousands of users over ~18 months and reduces cloud cost by shifting ad hoc workloads to Trino.
AI-written summary · about
chapters (5)
- 0:00Introduction to dual query engines
- 5:52Technical architecture overview
- 15:11Performance and governance lessons
- 21:33Observability and cost optimization
- 24:20Quick demo and Q&A
related talks
20mJourney of Building a Modern Data Prep Tool...
47mJOINs Are Slowing You Down: Why Spark and Trino Won’t Replace Your DWH
04mInterview
28mInnovations in AWS analytics: Data processing
01h 07mAn Overview of the Starburst Trino Query Optimizer
51mFast data processing in a data lake with SQL
51mPresto Open Source SQL Engine for Massive Scale Data Lakehouses
32mPromoting a data driven culture in a world of microservices