Distributed Systems: In the Daily Life of the Data Engineer
summary
When vertical scaling becomes too costly or unreliable, the talk explains moving to distributed systems and the resulting challenges: CAP tradeoffs, latency, data shuffles, and slow cluster startup. It demonstrates with a Bean Data Café example ingesting CSV into partitioned HDFS/Cassandra, running Spark (driver/executors) and Kafka (topics/partitions) to show Kafka partition lag and commit‑log offsets, compares storage formats (Parquet, Avro, Protobuf) and open table formats (Delta, Iceberg, Hudi), and analyzes consensus algorithms (Paxos, Raft, Zab) plus the performance cost of Spark shuffles and Raft replication.
AI-written summary · about
related talks
25mSPARK AND AKKA FOR BIG DATA SYSTEMS, IN PRACTICE
30mArchitecting Applications With Multiple Open-Source Big Data Technologies
29mBig Data Analysis
04mEvan Chan Interview
19mKeynote Panel Discussion: SMACK in the Enterprise
41mIntroduction to distributed systems
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka
52mSolving Analytical Problems using Apache Spark