The Perils of Large Files
summary
Shows concrete failures and fixes: reading a 128MB PDF into a long‑lived GenServer holds reference‑counted binaries and raises memory by ~166MB unless the process hibernates or caller reads the path; hashing with a naive read balloons memory but switching to FastStream with 8MB chunks keeps memory low; fetching a 12MB file via httpc can spike ~300–350MB, solved by a streaming receive loop and direct file writes or using httpc raw mode; recommends recon binary_leak, telemetry and soak testing to catch leaks.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:37TL;DR on PDF files • PDFs can be variable in size (20KB to 1+GB) Can be extremely complex/expensive to render
- 3:49Danger! • Memory starvation • Memory leaks • CPU/O starvation when operating on the file contents
- 5:28Out of scope for this talk • A one-size, fits-all solution A packaged library and/or large snippets
- 6:19Reading files • We have a file-system based cache store . Read the file by name • Process controls filesystem location
- 13:52Processing files . Given a file, we want to calculate its sha256 sum
- 19:21Writing files . Given a URL, fetch it and write the response to a file
- 27:08Limitations of raw mode Without intermediary process, we need to manually clos the io device with File.close/1 Only works on the same node
related talks
20mFrom Bytes to Brilliance : Optimizing Large File Delivery in REST APIs
50mWe Must Save the Last Giga of RAM
24mDemystifying Memory Leaks in JavaScript
15mStoring your data in kernel space: an excellent bad idea!
30mProcessing large JSON files without running out of memory
39mMemory Efficient Java
27mTwo memory leak stories in the story of flipkart.com
33mA Deep Dive into Binaries