How to help BI & AI people as a software engineer?
summary
Recommends specific fixes: grant query access via a read replica or nightly DB dumps, secure with VPN/SSH/IP restrictions, and stop sharing CSVs; provide a writable playground DB (RDS/Redshift/BigQuery) and a job runner like Jenkins or Rundeck so analysts can schedule Python/SQL transforms; expose a simple API to push model scores back into the product; implement an event-bus or cheap blob-based data lake (S3 buckets of nightly tables) and “log everything” for future ML needs; CTO or a tiny data-engineering team owns prioritization and integration.
AI-written summary · about
chapters (13)
- 0:00Intro
- 2:03Collaboration gap
- 5:20The problem
- 7:42Give data access
- 10:29Messy data
- 12:30Backend databases
- 13:07Writing data
- 15:15Running things
- 16:42Backchannel
- 18:16Hipster technologies
- 20:16Data lakes
- 22:48How to solve this problem
- 24:51Answering your questions
related talks
30mData Engineering building your BI infrastructure from scratch
38mKeeping pace with data infrastructure so it doesn't outrun you
35mBuilding Data-Intensive Teams
39mHelp Me, Help You: Supporting Your Data
32mMaking Happy Modelers: Build and Maintain...
30mInfrastructure scale issues
33mAI with a DevOps mindset - experimentation, sharing ..
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications