Applied Science Fiction: Operating a Research-Led Product
summary
Research-led AR pipelines use Buildkite + Kubernetes job CI to run unit and slow functional tests, with CI doing 5–10 canned work units and a weekly 14-hour full-corpus run for release gating. Modules are independent Celery workers on RabbitMQ, built into container images by a small k8s operator + KEDA autoscaling so every version can run side-by-side on separate task queues. Other specifics: structured logs via python structlog into Loki, retries and timeouts per work unit, static analysis adopted, test weeks quarterly, and user data anonymized before entering corpuses.
AI-written summary · about
chapters (23)
- 0:00THE ADVANCED COMPUTING SYSTEMS ASSOCIATION
- 1:17PIPELINES
- 1:37BUILDING GUARDRAILS (Most) researchers are not engineers And don't want to be engineers Build safe paths to move forward Strong mutual respect for specialization Progress with confidence and quality
- 2:53WRITING TESTS? Functional tests Easy harness Numeric metrics Unit tests, as applicable and comfortable
- 3:43CODE REVIEW Deputize the most technical Code quality, obvious performance issues, test coverage Science review: I stay out of it
- 4:08STATIC ANALYSIS » Pre-Commit: tool runner Black: code formatter Isort: more code formatting Flake8: code quality Overall well received
- 5:24DEPLOYMENT Ludicrously Automated Custom operator CI deploys on tag or main
- 6:34PARALLEL INSTANCES New release, new instance Latest builds update in place Old versions are left time-locked Unless there's a critical issue Pipeline definitions pin module versions
- 7:19AUTOSCALING Running every version forever is $$$ We already have a queue driven architecture Autoscaling!
- 7:49LATEST Work collisions Like an unpinned dependency Needed for corpus runs Overall positive
- 8:27PIPELINE DEFINITIONS Declarative is nice DSL is more realistic Compilable DSL is best
- 9:06DEFAULTS ONE-OFFS
- 10:11RUNTIME SUPPORT Helper libraries Format conversion Logging and metrics Common algorithms Asset sync, in and out
- 10:55NO REALLY, RETRIES! CELERY_ACKS_LATE Queue based retries Pipeline based retries
- 11:14TIMEOUTS Trust with retries but verify with timeouts Nested timeouts Work unit Individual steps If it has a timeout, it should also have a metric
- 11:42ASIDE: STRUCTURED LOGGING You already know this is good But really, it is logfmt is a nice mix of parsing but also humans Context variables that attach to every log line ts=2022... level=INFO msg="Hello world" run_id=6362
- 12:15SOCIAL SUPPORT Office hours Brown-bag talks Testing weeks
- 12:51TESTING WEEK No new features or fixes All quality all the time Great place to introduce new tools Gets everyone talking about tests/quality
- 14:08CORPUS MANAGEMENT Production edge cases? Personal info (PI) Easy copy tools w/ anonymization Paid testers whenever possible
- 15:07OPEN ISSUES
- 15:33LICENSING What is safe to use Experiments vs. production I am not a lawyer, talk to your legal team Internal training
- 16:07Engineering tools work for research too As long as you build them well DAGs are your friend New versions as new deploys Retry failures everywhere » Observability is for everyone Plan together, succeed together
- 16:37BE KIND
related talks
26mManufacturing Alerts To ReActive Agents: Self-Evolving AI Agents on K8S for Predictive…
22mDevOps in Data Science: What Works and What Doesn’t
27mFrom Experiment To Enterprise: Scaling an AI Agent for Code Review
20mDeveloper Productivity in the Age of GenAI
15mResearch to Reality: Bringing Frontier ML Research to Production
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case S...
37mBuilding AI4S Systems for Scientific Research on Kubernetes
27mGen-AI at Scale: Simplifying Orchestration of Healthcare Applications Across Multi-Cluster Environme