Scaling data from the lake to the mesh via OSS

on YouTube ↗29m

" summary

Claim: replace slow monolithic data lakes with a distributed read model that materializes entities at the edge and serves them via an API mesh. Implementation uses Fastify, GraphQl, Kafka streaming into an ephemeral entity store in elastic search fed by Kafka node, TypeScript codegen and graphql-compose-elasticsearch, CQRS for read/write separation, and a kuma service mesh with federated GraphQL. The stack produces predictable, low-latency reads and simplifies consumer schemas.

AI-written summary · about

" chapters (6)

1. 0:00Start
2. 11:35What about DATA?
3. 21:25npm i fastify-cli
4. 23:21npm i fastify-elasticsearch
5. 24:14graphql-compose-elasticsearch
6. 28:13@apollo/federation

" related talks

• 57mOur DataMesh Journey: data architectures for microservicesCarlos Saona-Vázquez · Spring I/O 2022
• 33mThe advent of the open data lakeJulien Le Dem · AI By the Bay 2025
• 47mModern Data Architecture: Breaking the Big Data MonolithDavid Ostrovsky · Build Stuff 2022
• 51mImmutable data stores and CORS for safety, flexibility and profitSidney Shek · Scala Days Berlin 2016
• 42mTransform your data approach: Develop a modern data strategyBlair Layton, Saeed Barghi · AWS Summit Singapore 2022
• 37mKafka Based Global Data Mesh at WixNatan Silnitsky · DevDays Europe 2022
• 52mFast but not Loose: Typesafe Clients in a Distributed Service ArchitectureEric Bowman · GOTO Aarhus 2013
• 29mScaling the Data Access by Leveraging GraphQL Federation for Service to Service CommunicationSerhii Korin · GraphQLConf 2023