How We Scaled Catalog Ingestion to Hundreds of Thousands of Entities
summary
HP Inc. describes ingesting hundreds of thousands of entities into Backstage by building incremental entity providers that page and back off, adding three DB tables to track ingestions, and moving all HTTP calls into providers to front-load async calls. Results reduce processing lag from hours or days to ~20 minutes in production and as low as 7 minutes in dev. Works with Kubernetes, Postgres, GitHub rate limits; requires paginated stateless APIs and more DB storage.
AI-written summary · about
chapters (10)
- 0:00The scale of HP's challenge
- 1:52Backstage and software role
- 5:39Problems with default methods
- 8:37Limitations of entity providers
- 12:23Performance bottlenecks
- 17:11Impact of async operations
- 20:20Incremental entity providers
- 28:16Optimizing processing loops
- 32:21Results and performance gains
- 35:12Lessons learned and next steps
related talks
26mAn Easier Path to a Rich Catalog
29mAutomatically generate Backstage entity descriptors for easy adoption
27mIngestify: Rethinking Ingestion for Complex Data
49mA Front Row Seat to Backstage Adoption
24mScaling to Billions of Internet Observations per Day
48mBuilding Petabyte-Scale PostgreSQL Deployments
22mIt’s all about the db - a journey to accommodate 100k events per second
24mGrowing Pains: Taming the Post-Startup Stage With Backstage