IGNITE: Maxime Boisvert - DevOpsDays MTL 2019
summary
Shopify replaces direct writes to one Elasticsearch cluster with Kafka queued writes and builds cluster replication by consuming the same Kafka topic into multiple ES clusters bootstrapped from snapshots so search survives a data center outage. Failover evolves from a slow pull request swapping hardcoded IPs to changing CNAMEs, then a Slack spy command that creates the PR and an instant spy endpoint set response, with Google Traffic Director and Route 53 automated later. System runs for maintenance, migrations and incidents; next step is fully nonhuman failover.
AI-written summary · about
related talks
42mPlanning To Fail
41mScaling Reliability Engineering with Tools
48mDevops@Scale
29mBlack Friday - Lessons in Resiliency and Incident Response at Shopify
40mKafka, DevOps...And Resilience for All
31mHow We Went From Two Major Outages to 99.98% Reliability in 6 Months
15mKeynote: Push It to the Limit: From Canary Deployments to Canary Clusters
10mLightning Talk: A Culture of Resiliency through Chaos