Bootstrapping Microservices using Akka, Kafka ans Spark

on YouTube ↗55m

" summary

Bootstraps microservices by turning Kafka into the canonical event store and replayable source of truth, using Kafka as source of truth, Avro + Confluent Schema Registry for evolvable schemas and log compaction to speed bootstrapping. Built a Scala/Akka gateway called Hydra (Scala/Akka) that validates/ingests JSON→Avro into Kafka, and a config-driven Hydra-Spark DSL that compiles into Spark Streaming jobs to upsert materialized views (Postgres) and write derived topics. Runs on a three-node Hydra cluster, handles roughly 500–1,000 msgs/sec, and is open source.

AI-written summary · about

" chapters (11)

1. 0:00Start
2. 2:14MONOLITHIC APPS
3. 5:25THE SERVICE DATA MISMATCH
4. 9:33MUTATION VS. FACTS
5. 12:29A MESSAGING SYSTEM BASED ON DISTRIBUTED LOG SEMANTICS
6. 13:47REPLICATION IS ABOUT RESILIENCY
7. 14:19THE LOG IS A LINEAR STRUCTURE
8. 14:48MESSAGES CAN BE REPLAYED
9. 15:25LOG CLEAN UP POLICY: DELETE
10. 15:50LOG CLEAN UP POLICY: COMPACT
11. 24:36INGESTION REPLICATION PROTOCOL

" related talks

• 41mReducing Microservice Complexity with Kafka and Reactive StreamsJim Riecken · Scala Days Berlin 2016
• 33mKafka Microservices with Kafka plus Akka and Kafka StreamsBoris Lublinsky · Scala Days Berlin 2018
• 44mKafka based Microservices with Akka Streams and Kafka StreamsDean Wampler · Scala Days New York 2018
• 59mImplementing Microservices with Scala and AkkaVaughn Vernon · Scala Days Berlin 2016
• 41mApache Kafka + Apache Mesos = Highly Scalable Streaming MicroservicesKai Waehner · MesosCon Europe 2017
• 48mThe three paths of Scaling with AkkaFran Perez · Scala.IO 2017
• 52m10 LESSONS LEARNED FROM USING KAFKA IN 1000 SCALA MICROSERVICESNatan Silnitsky · Scalar 2020
• 11mSpinning up micro-services using Ruby/KafkaAnkita Gupta · Red Dot Ruby Conference 2017