How Data Architectures Break and How to Fix Them
summary
The talk examines failure modes of distributed databases and cloud-based big-data architectures (EMR, Hadoop) and practical fixes. It explains when distributed databases and EMR/Hadoop become unreliable, why some databases are unsuitable in certain scenarios, and argues for migrating Kafka-consuming event processors from EMR to small Python and Go processes with separate orchestration, noting that Airflow can simplify and make data architectures easier to manage.
AI-written summary · about
chapters (4)
related talks
02h 18mDistributed Systems: In the Daily Life of the Data Engineer
31mData Unleashed: A Developer's Perspective on Navigating the Architecture Maze
58mTales of Data Architecture Evolution
39mModern Event-driven Architecture: Adopting Data in Motion
41mModern Data Architectures – What They Are and How to Implement?
40mThe Changing Face Of ETL: Event-driven Architectures for Data Engineers
31mZero to data infrastructure
49mWhen Performance, Scalability and Robustness Upset Your Architecture