Big Data With Open Source Solutions
summary
Automattic runs a Hadoop HDFS backed data lake on bare metal with Spark for ETL, Trino for low‑latency SQL, Yarn as the cluster resource manager, Airflow orchestrating about 4k Airflow jobs, and Superset/JupyterHub for access. Open source removes vendor lock and becomes cheaper at scale but forces engineering work: Meltano connectors prove fragile, FiveTran gets costly, and custom HDFS support for Airflow/JupyterHub is implemented and maintained by the team.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 3:23Open source culture at Automatic
- 4:51Hardware and infrastructure stack
- 6:17Data storage layer
- 9:25Hadoop and HDFS
- 11:10Hive as SQL interface
- 12:37Spark computation engine
- 13:21YARN resource management
- 14:22Airflow data pipelines
- 16:06Trino low-latency SQL
- 17:37Meltano data integration
- 20:26Access layer tools
- 22:37UI/UX challenges
- 23:40Jupyter Hub notebooks
- 24:24Superset BI visualization
- 26:14Additional tools overview
- 27:18Compatibility and maintenance
- 29:28Summary
- 31:55Q&A
related talks
29mOpen Source Powers The Modern Data Stack
06mOpen by design, open for data
06mDriving enterprise open source adoption, from data lake to AI
06mDemocratizing Big Data and Hadoop-Based Platforms and Solutions
48mUsing Open Source Tech to Swap Out Components of Your Data Pipeline
34mAdvance Auto Parts and Open Source
05mA New Approach to Big Data
10mThe Future of Open Source