Scaling data from the lake to the mesh via OSS
summary
Claims a practical pipeline that cuts read latency from hours to near real time by streaming source-of-record changes into a federated query layer. Combines Fastify, Kafka + Argo/Brigade workers, Elasticsearch as the materialized fast store, TypeScript-generated GraphQl schemas and Apollo Federation behind a Kuma service mesh. Shows automated schema generation, fewer bespoke services, and an implemented demo that routes queries through the gateway and serves fast, eventually consistent reads.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 2:20What happened with data
- 8:00A better way
- 13:30Kafka
- 16:36Elasticsearch
- 20:20Typescript
related talks
31mFrom Warehouse to Lakehouse: Data at Rest
25mMoving Data From Graph to the Warehouse, A Configuration Driven Approach
28mMoving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns
29mData Acquisition: Overcoming Challenges
34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics