GitHub Archive - Processing and analyzing the billions of events generated by the w
summary
GitHub Archive data from 2011 onward scales from hundreds of gigabytes to terabytes and the project plans a Petabyte of data import. Data lands on Amazon S3 and imports use bulk-optimized insert settings, shard and replica tuning, and tuned refresh intervals. The pipeline experiments with elastic search features, Presto for SQL planning, and a CrateDB demo app (Angular) that runs most queries locally but whose live demo fails due to network/connectivity issues.
AI-written summary · about
chapters (10)
- 0:00Introduction and welcome
- 2:22GitHub Archive overview
- 7:13Database scalability challenges
- 13:14Elasticsearch and Split startup
- 20:24Query execution and aggregation
- 26:01Data replication and sharding
- 31:07Full-text search and SQL compatibility
- 35:09Live demo setup
- 39:48Real-time analytics and use cases
- 44:20Closing and resources
related talks
48mHow to Analyze Terabytes of Data from GitHub Archive at High Speed
35mHow to Analyze Terabytes of Data from GitHub Archive at High Speed
01h 03mWhat can we learn from 750 billion GitHub events and 42 TB of code
14mGitHub's Data Story: Open Source at the Home for Open Source
01h 04mWhat can we learn from 750 billion GitHub events and 42 TB of code
41mStep By Step Guide to Building an Application Analytics System
19mLog Events @ Twitter: Challenges of Handling Billions of Events per Minute
44mBest practices for analytics and generative AI on AWS(ANT302)