Ten years of building open source standards.
summary
Story of three concrete projects and what worked: a columnar on-disk Parquet file format started 2012 with Twitter and Cloudera merging designs, enters Apache 2013 and graduates 2015 and becomes the de facto storage for every major warehouse; an in-memory Arrow zero-copy format spun out of Drill to enable vectorized, cache-friendly execution and fast JVM/C++ interop; an OpenLineage lineage spec peels metadata out of Marquez to cut N^2 integrations to a single standard and sees rapid adoption by Microsoft Google Netflix and others.
AI-written summary · about
chapters (9)
- 0:00Introduction to ten years
- 1:11Birth of Parquet project
- 12:58Lessons from Parquet growth
- 17:15Introducing Apache Arrow
- 20:51Arrow’s exponential growth
- 23:45Open Lineage for data reliability
- 28:58Open Lineage’s adoption
- 31:51Aligning community incentives
- 34:00Key takeaways and conclusion
related talks
56mWelcome & State of Open Source Databases
45mState of the Duck
05mA New Approach to Big Data
09mData: Open for Good and Secure by Default
06mOpen by design, open for data
40mInnovating in Open Source in Your Enterprise
27mDave Stokes talks open - Stocking the flames of open source - Episode 1
36mOpenbytes Builds Open Dataset Standards and Schema Aiming to Bring Transformational Changes to AI by Making Open Datasets More Available and Accessible