Build your open data lakehouse with Dremio and AWS (PRT085)

on YouTube ↗13m

" summary

Shows a Dremio + AWS reference architecture that stores Parquet/Iceberg on Amazon S3, uses Glue or Dremio Arctic with Project Nessie as the metastore, and runs query engines against open tables so data is not copied. Claims ACID table operations, time travel and DML (upserts/merges) on Iceberg, materialized Reflections delivering sub-second queries on billions of rows, and cites Apache Arrow (about 60 million downloads per month). Notes typical two-tier setups create ~9 copies per dataset; Dremio Cloud offers a free SaaS trial.

AI-written summary · about

" chapters (5)

1. 0:00Introduction to Dremio
2. 1:25Data architecture evolution
3. 5:11Defining the data lake house
4. 7:38Reference architecture
5. 10:38Importance of open systems

" related talks

• 45mBuilding an Open Data Lakehouse using Apache Spark, Apache Iceberg and DremioMike Flower · SQLBits 2023
• 45mBuilding an Open Data Lakehouse using Apache Spark, Apache Iceberg and DremioMike Flower · SQLBits 2019
• 25mBuild open format data lakehouseSatish Mane, Rajeev Jahswal · Conf42 Cloud Native 2023
• 01h 04mQuery Optimization and Acceleration at DremioSteven Phillips, Vivekanand Vellanki · Vaccination Database Talks (Second Dose) 2021
• 47mConstruyendo Lagos de Datos y Analíticos en AWS + Datos AbiertosAngelo Carvalho · AWS Public Sector Summit Mexico City 2020
• 25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant AnalyticsMark Lyons, Jonny Dixon · Big Data LDN 2022
• 37mBuild a Modern Data Platform on AWSArtsiom Yudovin · Big Data & RPA Conference Europe 2024
• 56mUnderstanding and achieving a modern data architectureChanakya Damarla · AWS Summit San Francisco 2022