Achieving Concurrency in Streamlit with a RQ scheduler
summary
Claim: Streamlit apps become responsive under concurrent users by pushing long jobs into Redis Queue (RQ) workers while the main app stays interactive. Implementation uses a local Redis server, an RQ queue, spun-up worker processes, and a scheduler that enqueues user-defined function via UI calls. Demonstration shows multiple tasks submitted nearly simultaneously complete concurrently, and confirms limitations: multiprocessing cannot write back to Streamlit UI and Tasklit only runs command-line jobs. Also uses Streamlit’s fragment_run to update status without full reruns.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 2:10Streamlit concurrency challenges
- 5:01Streamlit concurrency problems
- 7:32Streamlit RQ scheduler proposal
- 9:23Multithreading multiprocessing Taset
- 12:57Redis Q advantages
- 15:33Multithreading demo
- 17:39Multiprocessing limitations
- 18:43Taset limitations comparison
- 19:27Multithreading needs queue
- 21:20Redis Q architecture benefits
- 23:10Streamlit RQ code structure
- 26:29Streamlit RQ features
- 27:12Ordering system demo
- 31:28Conclusion Q&A
related talks
42mStreamlit: A Faster Way to Build and Share Data Apps
01h 58mHow to create beautiful interactive GUIs and web apps
28mBeyond Prototyping: Building Production-Level Apps with Streamlit
09mBuild Interactive Web Apps with Python: Streamlit and Supabase
18mStreamlit: The Fastest Way to build Data Apps
27mReal Time Shiny Applications
18mDon’t let long running tasks hang users: introducing IPC for Shiny
44mMaking apps seem faster without optimizing