A tale of futures, actors & pipes: using Akka for large data migration
summary
Claim: an Akka actors Futures and Pipes pipeline moves a startup’s data (over 3 million users) in a single big‑bang weekend migration by paging SimpleDB, fetching files from S3 YouTube SoundCloud and other clouds, and routing work through supervised pools and round‑robin routers. Concrete lessons: rate limits forced throttling, blocking I/O causes OOMs so use dedicated execution contexts, capture sender when piping futures, and incremental tests plus tooling for resumes and data‑quality checks make it actually succeed.
AI-written summary · about
chapters (4)
related talks
58mLife with actors: experience report
41mAkka in Production: Our Story
55mExperience report on using Akka in production: pitfalls and rewards
51mComposing high performance process workflows with Akka Streams
55mHow to fail at building a reactive information system? — MNT postmortem
52mThe best is yet to come - State of Akka in 2017
01h 01mCurrent Methods for Building Data Pipelines
59mDrinking a river of IoT data with Akka.NET