Cooperating Gault Tolerant Habitat #cheffriends
summary
Claims a working pipeline built with Habitat packages, using Postgres staging and Redis Streams streaming module as the system of record; a simulator injects a 20% failure injection rate while producing ten test records. Components: API stages JSON into Postgres, a streamer moves committed batches into Redis, multiple identical consumers checkpoint and deduplicate by primary key for idempotence. The demo UI breaks but the data story holds: duplicates are ignored and consumers recover from crashes, proving the approach.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:53ACID guarantees and design
- 3:23Idempotence and data
- 4:47Habitat transducer overview
- 5:50Live demo setup
- 7:14Process types and failures
- 8:18Idempotence in demo
- 10:03Habitat visualization and consistency
- 12:51Demo recap and resilience
- 13:32Failure injection and recovery
- 14:59Checkpointing and crash recovery
- 15:39Summary
related talks
05mIgnite: Introduction to Habitat
37mIntelligent Serverless and Scalable Real-Time Data Pipeline using Kinesis, Fargate and CFN
21mLive-coding demo
42mTrick or Treat Your Pipeline: A Haunting Tale...
40mFrom Tremors to Saving Lives: Serverless Kafka Arch for Earthquake Notifications
25mPersistent and performant ingestion using Data Prepper
14mRobust R Deployments: Building a Pipeline from RStudio to Production
42mBuilding real-time serverless data applications