How Streaming Helps Your Staging Environment and Sandboxes Stay Up To Date
summary
Shows a working pipeline that streams anonymized API requests/responses into Apache Flink and transforms them into an evolving mock server by retaining only a compact state and applying diff-based updates so snapshots can be replayed for staging. Uses Kafka-style ingestion, deploys the mock as a Flink stateful functions instance, handles throttling and persistence, and reduces the need for multi-year retention while enabling rewindable windows for debugging issues that otherwise appear only after tens of millions of events.
AI-written summary · about
chapters (12)
- 0:00Intro
- 1:48Stories from the dark side
- 6:07But staging environments had their problems...
- 7:01Containerization helps mocking
- 7:32But there are still issues
- 12:09Benefits of in-memory mocks
- 12:47Downsides of in-memory mocks
- 15:04Benefits of mock servers
- 16:53How mocking with streams solves the mock server problem
- 19:02Mocking with streams - Kafka
- 20:58Flink solves the state problem in service mocking
- 24:41Meeshkan: Your partners in testing using Flink
related talks
31mTale of Stateful Stream to Stream Processing
37mBuilding and operating large-scale streaming applications at King
38mTaking away customer friction through streaming analytics
42mApache Flink Worst Practices
36mStream All The Things!
38mDeploying Stateful FaaS on Streaming Dataflows
43mStream Processing and Applications in the Modern Age
36mApache Flink in the Cloud-native Era