Fireworks - lighting up the sky with millions of Sparks
summary
Salesforce Einstein runs Spark pipelines for each customer and scales by turning per-app Azkaban clusters into a template-driven multi-tenant workflow service that hides Yarn and Spark configs. Early stacks use Azkaban + YARN + Spark and reach about 100k spark jobs per day and roughly half a billion predictions daily. The service enforces weaker SLAs, governor limits and idempotent templates so the platform can control resource allocation, swap internals later, and reduce customer-caused failures.
AI-written summary · about
chapters (8)
- 0:00Introduction to Einstein AI
- 3:20Multi-tenant predictive applications
- 9:09Building first applications
- 16:09Designing compute service
- 24:07Scaling with less freedom
- 29:12Product drives scalability
- 35:52Template-driven workflows
- 38:40Takeaways and Q&A
related talks
25mSPARK AND AKKA FOR BIG DATA SYSTEMS, IN PRACTICE
19mApache Beam pipelines at 100TB+ scale using Apache Spark
31mManaging Multi-Cloud Apache Spark on Kubernetes
36mIn-memory data pipeline and warehouse at scale
28mAkka - Enterprise Agentic AI with Tyler Jewel
01h 08mDystopia as a Service
19mKeynote Panel Discussion: SMACK in the Enterprise
14mManaging decisions with Amazon Kinesis for better outcomes