What can we learn from 750 billion GitHub events and 42 TB of code
summary
Five years of GitHub metadata and 750 billion GitHub events plus 42 TB of code live in public datasets and get analyzed with BigQuery and GHTorrent. Queries join Hacker News and Stack Overflow, count distinct stargazers (revealing freeCodeCamp’s duplicate-star inflation), map pushes per capita (Iceland tops), detect 448 files copied from StackOverflow, show Go’s enforced tabs, and quantify that a proposed Go change would help ~2,000 repos while TLS capitalization changes would break 685 — all runnable as SQL with JS UDFs and static analysis at scale.
AI-written summary · about
chapters (14)
- 0:00Start
- 0:12Felipe Hoffa
- 4:01Github Archive
- 7:01Google Data Studio
- 8:51Top Projects by Stars in 2016
- 9:01Bigquery Ui
- 15:15Vs Code
- 19:34How To Start an Issue
- 36:32Stack Overflow
- 37:53How To Randomize a List
- 39:20Requesting Features
- 53:43What Happens with Your Sensitive Data
- 54:18Open-Source Tools To Capture Data into Bigquery
- 55:27Ingestion Process and Bigquery
related talks
01h 04mWhat can we learn from 750 billion GitHub events and 42 TB of code
14mGitHub's Data Story: Open Source at the Home for Open Source
48mHow to Analyze Terabytes of Data from GitHub Archive at High Speed
30mData Tales from an Open Source Research Team
45mGitHub Archive - Processing and analyzing the billions of events generated by the w
35mRegulations, Ethics And Messy Data: Tales From An Open Source Analyst
27mCommunity Data is Not Community Metrics
34mQuantifying the growth and evolution of open-source communities with Apache Superset