CodeCommons_ Towards transparent, richer and sustainable datasets for code generation model training
summary
CodeCommons builds a metadata-rich code corpus on top of Software Heritage and claims an ingestion pipeline that will catch up ~150 million GitHub repositories. The implementation runs on an HPC cluster using Kafka and RocksDB for streams and indexes, a Redis-style queue and object storage, and a custom dduplicator component that exports OC-format graph objects. Indexes swell to 1.4 terabytes, an index corruption is repaired, and the pipeline currently runs and is being benchmarked to finish ingestion in a few months.
AI-written summary · about
chapters (14)
- 0:00CodeCommons project intro
- 2:08Software Heritage’s mission
- 3:12Enriching code with metadata
- 4:56Infrastructure contributions
- 7:45Ingestion pipeline overview
- 10:59HPC constraints
- 13:32Deduplicator deep dive
- 17:51HPC challenges and solutions
- 19:20Ethics transparency goals
- 20:22Next steps and resources
- 21:23Q&A: Motivation and infrastructure
- 23:56Q&A: Ingestion timeline
- 24:38Q&A: Accessibility and UI
- 26:27Q&A: Query capabilities
related talks
44mSoftware Heritage: DevOps challenges to preserve our software commons
40mSoftware Heritage
28mDelivering AI on Code: Live Demo of source{d}
27mIs InnerSource Commons good for open source?
34mThe web of commons
36mArkisto: an open-source, standards-based framework for digital preservation
15mKeynote: Common Corpus: Opening Data for Building Open Source LLMs
15mKeynote: Pre-training and Fine-tuning of Code Generation Models