Promoting a data driven culture in a world of microservices
summary
Hudl consolidates logs and databases into Amazon Redshift and opens warehouse access to all 430 employees, runs 24 TB and 100B+ rows, and spins up data engineering and data analytics squads. ETL runs on Luigi, Jenkins and EMR, queries and dashboards serve through redash hosted on EC2, and a searchable data dictionary plus derived tables simplify common joins. Training certifies ~100–150 people, monthly activity shows 194 unique query users and 14,000 ad hoc queries, adoption rises but poor queries and occasional performance remain challenges.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:07Huddle and its product
- 2:32Huddle’s data journey
- 4:42Data engineering teams
- 5:23Redshift for analytics
- 6:47ETL workflows
- 8:56Reliable data pipelines
- 11:21Team access to Redshift
- 13:09Analytics team mission
- 14:34Data philosophy challenges
- 15:58Education and training
- 17:44Data dictionary
- 19:07Derived tables for queries
- 21:13Automated reports with Redash
- 22:20Metrics via Slack
- 24:06Key takeaways
- 25:33Tools review
- 26:11Q&A: SQL and automation
related talks
30mData Engineering building your BI infrastructure from scratch
40mData Intelligence Platform on AWS
17mDemocratising data analytics
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales
23m‘Samen Slimmer met Data’ – A company wide Data Transformation
48mData warehousing reinvented for today’s needs
31mBuilding large scale, transactional data lakes with Apache Hudi
37mBuild a Modern Data Platform on AWS