Journey of Building a Modern Data Prep Tool...
summary
Workday Prism Analytics builds a data‑prep engine on Apache Spark that runs two apps: always-on interactive for sampled, low‑latency queries and on-demand batch for full pipelines. It compiles UI-defined transforms into Spark SQL/Catalyst plans, uses Tungsten codegen for 100+ custom functions, and leverages the Data Sources API for partition elimination and incremental analysis. Practical fixes include applying a 100k intermediate limit on joins and correlated, bucketed sampling (scan ~2–10x requested rows) to avoid empty or explosionary results.
AI-written summary · about
chapters (10)
- 0:00Introduction to the talk
- 1:50What is Prism Analytics
- 3:38Data prep vs traditional ETL
- 6:27Why Spark for data prep
- 8:14Spark workloads in Prism
- 10:00Challenges in data prep
- 12:27Handling empty join results
- 15:19Correlated sampling for joins
- 17:54Other challenges and closing
- 18:23Q&A session
related talks
15mTransforming raw data into trusted analytics
27mHarnessing the Power of Dual Query Engines.
43mData warehousing reinvented for today's needs
30mData Engineering building your BI infrastructure from scratch
39mBuilding a Modern Data Warehouse Without Breaking the Bank
17mDemocratising data analytics
21mPutting big data self-service analytics in the hands of business people
15mOptimizing Query Performance in Big Data Systems