MY AWFUL EX-DATASYSTEMS
summary
Covers concrete failures and fixes: Flume crashes on 4‑byte UTF‑8 emojis, MongoDB limits and alternatives, Spark hype versus real needs, and a legacy pipeline that read from ~200 DB instances and accumulated over 500 config files then reduced to <20. Describes replacing feedback loops with a single denormalized big blob and tiny metric reporters, making the pipeline deterministic and idempotent, preferring consistency over chasing edge case correctness, and the payoff of simpler systems that are debuggable and reliably runnable locally.
AI-written summary · about
chapters (5)
- 0:00Start
- 7:49Boring Software Needs Love Too
- 12:03The Backseat Driver
- 15:14The Loser
- 21:11You can do better than MongoDB
related talks
31mWhen the data levee breaks: Managing cognitive load in data-intensive projects
30mDesigning Resilient Data Pipelines
28mSoftware Engineering Skills for Data Engineers
44mTaking the plunge - a deep dive into streaming with fs2
37mData Swamp to Data Swimming Pool: Cleaning up the Mess
25mData Demons: Unleashing Chaos or Driving Performance?
50mThe Babel Fish Data: A Case Study
34mTaking the blame out of our data transformation