Blazing-Fast Directory Tree Traversal: Haskell Streamly Beats Rust
summary
Claims a Haskell program using Streamly and low‑level readdir(3) I/O outperforms Rust fd and matches or beats GNU find on a 60,000‑node Linux tree. Implementation removes extra stat calls, batches output into a 32k buffer, uses concatIterate and parConcatIterate for DFS/BFS and parallelism, and drops allocations from ~450 MB to ~20 MB. Serial optimized Haskell hits ~134 ms, chunked version ~144 ms, concurrent version falls to ~85 ms and is reported ~1.5× faster than concurrent fd while producing identical output.
AI-written summary · about
chapters (10)
- 0:00Talk overview and motivation
- 2:24Naive implementation analysis
- 6:46Incremental performance tuning
- 10:40Streaming and combinators
- 16:33Optimizing with batching
- 21:12Refining for maximum speed
- 23:37Concurrent traversal and benchmarks
- 25:51Comparative performance analysis
- 29:25Lessons and systems programming
- 33:53Q&A session
related talks
48mStreamly: Declarative Concurrency And Dataflow Programming In Haskell
55mDeforestation and Program Fusion
38mLightning talks
01h 01mWeb Programming and Streaming Data in Haskell - Part 2 of 2
59mWeb Programming and Streaming Data in Haskell - Part 1 of 2
01h 02mOptimising for fast builds with GHC
56mDeterministic Parallel Programming with Haskell
01h 02mHaskell for Java Programmers