Watershed: Get to zero carbon, fast
summary
Watershed runs monthly carbon-footprint parquet files (largest customer 17M rows, typical footprint ~750 MB) from Google Cloud Storage through hosted DuckDB, replacing a giant Postgres table. A local byte cache avoids re-downloading files, yielding roughly 10x faster than Postgres on comparable analytics, serving ad-hoc group-by queries with P99 < 1s and handling about 75,000 queries/day. Migration reduced storage and removed complex indexing and migration pain.
AI-written summary · about
related talks
50mFast Analytics On Big Data
41mOn Data: Sky above, sand below, peace within
17mBuilding Water Security From the Bottom Up by Leveraging Big Data
31mBeyond Pandas: lightning fast in-memory dataframes with Polars
01h 02mAccelerate data science and analytics with GPUs
19mA world of data
21mGPU-accelerated Hydrology Algorithms for On-prem Computation: Flow accumulation,(…)
18mA Toolbox for Scalable Spatial Data Science in the Cloud