Accidental DataOps: How we matured our organization with data, and how you can too!
summary
Begins with manual tarball pulls and grep/jq for alert data, spikes in customer alerts become a sales signal, then overengineers a Kafka-based pipeline with Go microservices but ends up relying on simple joins in Postgres and ad‑hoc Python/cron jobs. Teams expose filtered alert trends to sales, join HTTP logs with Salesforce for feature usage, and deploy MetaBase for SQL-driven self-service so non-engineers explore data. Real findings: only megabytes per customer per day, early wins from Postgres with JSON columns, tooling like JQ and cron+Python ad‑hoc jobs, and a deadweight Kafka and Go microservices layer. Challenges: governance, testing, productization.
AI-written summary · about
chapters (8)
- 0:00Start
- 7:32Analytics Hunting and Gathering
- 9:25Alert Volume by Customer
- 11:38The Second System Design
- 15:29Our DataOps Story
- 16:48The Emergent Design
- 24:03Spread the Word
- 31:50More Info
related talks
34mTales of Data Architecture Evolution
13mData for product
34mData Engineering Architecture
30mDevOps For Data Teams
47mData - The Land DevOps Forgot
38mData Secrets from a Platform Engineer
42mScaling Data at Ovoko: The Succesful Strategy
23mFrom Start-Up to Scale-Up: Building a Data Team at a High-Growth Global Ed-Tech Company