Democratization of Data
summary
When a company moves to microservices, independent teams create different logging and event formats, causing a ‘Babel’ of data. The talk explains building a centralized event collector used company-wide in four months: why a central event store is important, the team profile that built it, how it improved cross-team communication, and how change management drove adoption of a common log consolidation standard. Technologies mentioned include Java microservices, Hadoop, Hive, Spark, Cassandra and Elasticsearch.
AI-written summary · about
chapters (11)
- 0:00Start
- 4:55Data processing
- 7:04Blitzscaling or growing like crazy
- 8:17Building the data team
- 12:00Problem summary
- 14:20Skills
- 14:54Profile: superheroes
- 18:08Size: complexity
- 19:20Centralizer
- 20:00Architecture
- 22:11Productivity
related talks
57mOur DataMesh Journey: data architectures for microservices
49mTransforming Analytical Data with Event-Driven Architecture
48mIntegrating microservices by accepting the data dichotomy
02h 18mDistributed Systems: In the Daily Life of the Data Engineer
37mEvolution of Data Collection in the Datalake at Samokat Infrastructure
34mTales of Data Architecture Evolution
44mThe Changing Face of ETL: Event-Driven Architectures for Data Engineers
33mMicroservices, Events, and Breaking the Data Monolith with Kafka