How to Analyze Terabytes of Data from GitHub Archive at High Speed
summary
Shows a working pipeline that loads gharchive JSON lines, stores them in Victoria Locks, and runs real queries on a playground backed by a single Kubernetes pod with 32 GB RAM and 8 cores. Demonstrates 7 TB of raw events compressed to ~1 TB on disk, 1.7 billion events counted in <0.5 s, and import that ingests a day of data in ~1 minute so a year loads in roughly an hour. Queries and demos run live and complete.
AI-written summary · about
chapters (14)
- 0:00Analyzing GitHub Archive data
- 1:10Speaker background and GitHub Archive overview
- 3:01GitHub Archive structure
- 9:27Unix tools and grep
- 13:04Relational DB challenges
- 15:36Why specialized DBs
- 18:27VictoriaMetrics Logs intro
- 19:33Demo setup and benchmarks
- 24:56Live demo queries
- 34:45Advanced query examples
- 36:08Known issues and missing fields
- 39:02Key fields and query features
- 40:26Use cases and tips
- 40:49Q&A
related talks
45mGitHub Archive - Processing and analyzing the billions of events generated by the w
01h 03mWhat can we learn from 750 billion GitHub events and 42 TB of code
01h 04mWhat can we learn from 750 billion GitHub events and 42 TB of code
48mHyper Speed! When Big Data Blooms
31mA Picture is Worth a Thousand Words: Bring Data Visualization in Backstage
41mAnalyzing Terabytes of Data with Google BigQuery
39mWhen Rivers Speak Analyzing Massive Water Quality Datasets
35mPowering Interactive Data Analysis with Google BigQuery