Reliable Kafka-Lambda Data Pipelines with AsyncAPI and Specmatic

on YouTube ↗25m

" summary

Uses an AsyncAPI spec to generate XML payloads from AsyncAPI and post them to Kafka topics, spins the processing Lambda up locally with LocalStack, and captures the downstream message to validate reply schema and correlation ID. Tests are generated at runtime by Specmatic so no hand‑written tests are needed; a deliberate code change (wrong order status) immediately causes a failing contract check. Supports per‑function and end‑to‑end pipeline verification for early, actionable feedback.

AI-written summary · about

" chapters (11)

1. 0:00Introduction
2. 2:59Problem with Kafka-Lambda pipelines
3. 5:29Live demo setup
4. 8:04AsyncAPI explained
5. 10:14Specmatic contract testing
6. 12:52Testing Lambda changes
7. 14:15Q&A
8. 16:50AsyncAPI multi-protocol support
9. 19:49Executable Specmatic contracts
10. 21:12Contract-driven development
11. 24:01Benefits and enterprise use

" related talks

• 23mThe Enterprise Guide To Building A Data MeshSion Smith, Neil Avery · Big Data LDN 2022
• 31mOne Data Pipeline to Rule Them AllSam Kitajima-Kimbrel · PyCon AU 2017
• 28mBuilding Event Streaming Applications with Serverless KafkaRicardo Ferreira · Serverless Practitioners Summit 2019
• 42mEnd-to-end pipeline agilityLars Albertsson · Berlin Buzzwords 2024
• 53mNoLambda: A new architecture combining streaming, ad hoc, machine...E. Chan · Philly ETE 2016
• 12mServerless Stream Processing with Confluent Cloud, Kafka and AWS LambdaGeetha Anne · Moar Serverless 2022
• 43mBuilding Scalable and Extendable Data PipelineYaroslav Tkachenko · Craft Conference 2019
• 30mSMART DATA PIPELINES FOR AIŁukasz Żuchowski · Scalar Conference 2023