Global School Connectivity DataOps Platform
summary
An open source end to end DataOps stack manages connectivity and infrastructure for 2.1 million mapped schools with a goal of 6 million, and delivers daily telemetry for about 93k schools. The platform uses Giga sync for standardized ingestion and quality checks, Daxter for orchestration, Spark jobs for data quality, a Delta Lake on ADLS exposed via Trino and Delta sharing APIs, and DataHub + Superset for cataloging and analytics; two flagship apps (Giga Maps, Gear Accounts) already use the APIs with ~5k monthly users.
AI-written summary · about
chapters (7)
- 0:00Giga UNICEF mission
- 3:10Open source tools
- 9:33Scalable DataOps platform
- 14:51DataOps architecture
- 22:19Giga Sync ingestion
- 25:31Data Hub discovery
- 28:20Future decentralization
related talks
47mData - The Land DevOps Forgot
23mFrom Start-Up to Scale-Up: Building a Data Team at a High-Growth Global Ed-Tech Company
40mThe Tools Empowering 5000 Employees
39mThe campus as a living lab: Collaboration with data lakes
47mDataOps for AI-Ready Platforms
12mMy School Today: Crowdsourcing geospatial data for mapping school locations
53mData Mesh in Practice - Assuring Data Quality at Scale
29mOpen Source Powers The Modern Data Stack