DataEngBytes 2023 - PER-T1-04 - Nathan Glover
summary
Edge devices batch JSON into gzipped files on disk and use AWS Greengrass StreamManager export tasks to upload partitioned objects to S3; S3 object events feed SQS and a Lambda that copies to a scratch bucket and loads via a temporary Hive table into Apache Iceberg. Iceberg handles partition evolution, schema changes and time travel while compacting files to ~512 MB to cut GET costs; example math shows an end-to-end 10 TB scenario around $230/month (plus KMS overhead) and typical query latency ~1.5s.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:07Talk plan and ingestion focus
- 3:35IoT data ingestion patterns
- 8:12S3 cost efficiency
- 12:28Apache Iceberg benefits
- 18:04Pipeline to Iceberg
- 23:20Athena visualization
- 24:23Cost breakdown
- 27:12Q&A
related talks
21mDataEngBytes 2023 - MEL-T1-05 - Fabio Ramos
26mDataEngBytes 2023 - SYD-T3-01 - Scott Eade
26mArchitecting real-time transactional data lake
36mAn Exploration in Storing Telemetry in Cloud Object Storage
21mOptimize and manage data on Amazon S3
50mDesigning a scalable data platform based on Apache Spark/Iceberg
39mApache Iceberg Merge-On-Read: Streaming CDC
24mDataEngBytes 2023 - MEL-T3-03 - Akira Wang & Abdi Farah