When Rivers Speak Analyzing Massive Water Quality Datasets
summary
Claims that terabytes of sensor records become tractable by combining USGS Water Data APIs with Remote SSH in Positron and columnar storage. The workflow fetches national site indexes, async-ingests three years of four parameters into statewise Parquet partitions (three years → about 1.88 GB Parquet for the selected parameters, Pennsylvania shard ≈20M rows), handles -99999 sentinel values, and cuts local runtime from ~71 to ~33 minutes while noting API rate limits and that the ingestion scripts and notebook explorer live on GitHub.
AI-written summary · about
chapters (16)
- 0:00Rivers as data storytellers
- 2:35Rivers recording history
- 4:47USGS sensor network
- 6:56Fetching river data challenges
- 9:01Water quality data storage
- 11:13Key water quality parameters
- 13:21Async state-level ingestion
- 16:17Remote SSH in Positron
- 18:50Benefits of remote SSH
- 22:21Positron data explorer
- 25:19Interactive exploration pitfalls
- 28:31Positron vs VS Code
- 30:39Performance gains with SSH
- 33:29Scaling 30 years of data
- 34:16Conclusion and future
- 36:45Q&A
related talks
48mHow to Analyze Terabytes of Data from GitHub Archive at High Speed
03h 07mData of Unusual Size: Interactive Visualization
48mHyper Speed! When Big Data Blooms
10mData Rivers — The New Analytics Architecture
20mBatching vs. Streaming - Scale & Process Millions of Measurements a Second
47mAnalytics at the speed of Direct Lake
45mGitHub Archive - Processing and analyzing the billions of events generated by the w
36mAn Exploration in Storing Telemetry in Cloud Object Storage