What can we learn from 750 billion GitHub events and 42 TB of code
summary
Analysis of 750 billion GitHub events and ~42 TB of source uses BigQuery with public datasets like GH Archive and GHTorrent, SQL plus JavaScript UDFs and JSHint for static analysis. Concrete results include per-month unique-star counts (freeCodeCamp and TensorFlow comparisons), Hacker News spikes driving sudden star bursts, country-by-developer-per-capita rankings (Iceland, Nordic countries; Singapore notable), detection of 448 copied StackOverflow snippets with only 27% attribution, and a language-by-language tabs vs spaces vote showing Go favors tabs while Ruby and Java favor spaces.
AI-written summary · about
chapters (18)
- 0:00Intro
- 1:40What do we see
- 2:41Who wants to analyze GitHub
- 6:00How GitHub events started
- 7:41Google BigQuery
- 15:26Comparing projects
- 17:29Looking for stars
- 18:59Looking at other projects
- 24:23Text analysis
- 26:19Country analysis
- 28:36New Zealand
- 29:31Weather
- 33:59Code analysis
- 39:14Stack Overflow
- 42:15Go Query
- 44:17Static Code Analysis
- 50:44Questions
- 55:14Query Analysis
related talks
01h 03mWhat can we learn from 750 billion GitHub events and 42 TB of code
30mWhat can you learn from thousands of source files in Github?
48mHow to Analyze Terabytes of Data from GitHub Archive at High Speed
14mGitHub's Data Story: Open Source at the Home for Open Source
35mHow to Analyze Terabytes of Data from GitHub Archive at High Speed
19mTop datasets for top DevRel results
35mRegulations, Ethics And Messy Data: Tales From An Open Source Analyst
25mData Science for OSPO: OSS Trends Found from Developer Activities