Panel discussion - Melbourne
summary
Panelists report real outcomes: a production lake house in production ~1 year offloads Redshift and lets ML jobs read from S3 without impacting BI, an engineering program that unifies 50+ sources integrated for reporting, and Kafka-based streaming patterns used as a decoupling tool—Kafka streams + S3 for ML with schema-driven compatibility. They warn data mesh is organizational not just tech, stress fixing S3/privilege hygiene after breaches like Optus, and advise learning fundamentals over chasing transient tools.
AI-written summary · about
chapters (13)
- 0:00Welcome
- 1:52Data engineering success
- 5:06Elite data teams
- 6:49Data lake to lake house
- 9:58Data mesh principles
- 13:10Data mesh challenges
- 17:42Data mesh adoption factors
- 18:25Scaling infrastructure tenfold
- 20:10Data scientist to engineer
- 23:00Optus breach response
- 26:55Advice for new engineers
- 29:43Breaking data silos
- 32:12Closing
related talks
33mPanel Discussion
44mFireside Chat: Building The Lakehouse
47mExpert Panel: Current & Future State of the Art in Real-Time Data Infrastructure
54mThe Future of Data Science/ML at Scale: A Look at MLflow, Delta Lake + Lakehouse
59mPanel: Future of Data Services
01h 14mPANEL - Streaming.
43mData Lake beyond the Buzzword
41mOn Data: Sky above, sand below, peace within