Taking Node.js to The Ex-Stream: Stream to Scale
summary
A production microservice moves from fetching entire JSON results into memory to a streaming pipeline that reads from Presto (self-managed to avoid athena’s 5-concurrent-query limit), transforms JSON-to-CSV with node.js Streams, and writes to s3 while keeping memory near 21 MB versus 2.6 GB for the in-memory approach. Code uses stream pipeline + util.promisify for reliable error handling and demonstrates generator/async-iterator integration to scale to gigabytes or terabytes.
AI-written summary · about
chapters (15)
- 0:00Start
- 3:29Don't be afraid of Streams
- 4:47LOVE Streams
- 8:32In memory solution
- 12:24What are Streams?
- 17:58Readable Stream Example
- 19:05Transform Stream Example
- 20:15Streams Tips
- 23:14Back pressure
- 24:20ObjectMode option
- 25:18highWaterMark option
- 28:04Async Generator Function Creates Async Iterable Object Async Generator
- 29:13Async Generator and Async Iterator
- 29:29Pipeline API Supports Generators
- 30:39Summary • Take Node to the ex-Stream!
