Log analytics platform - build or buy: case study of Moonfrog Labs
summary
Decision point is clear: Moonfrog builds a self-managed ELK stack that ingests ~300GB/day, targets 99.90% uptime, and keeps frequently queried data for 8 days and hot retention for 90 days while moving older shards to warm/cold storage. The stack runs Elasticsearch on Spot instances, uses S3 as a hot backup with recover-on-demand, and aggressively tunes replica counts, JVM/heap and thread settings. Removing replicas and shifting to Spot plus hot/cold layout reduces operational cost by roughly 50% to 85% and yields a measurable cost/GB and recovery trade-off that actually works.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:55Our problem statement
- 1:32Our business requirements
- 2:36Choosing the right option
- 4:07ELK Stack Overview: Terminologies
- 5:52Our ELK architecture: Hot-Warm-Cold data storage infinite scale
- 7:40Our ELK architecture: Size and scale
- 8:39Optimizations we did: Logstash
- 12:04Optimizations we did: Application side
- 15:03Optimizations we did: Infrastructure side
- 15:22Optimizations we did: EC2 and spot
- 17:20Optimizations we did: EBS
- 19:03Optimizations we did: Inter-AZ data transfer
- 19:45Data backup and restore
- 20:34Disaster recovery
- 26:51Key Takeaways
- 30:25Reflection
related talks
46mBuilding a scalable logging platform
41mScaling Observability: From Chaos to Clarity with Open Source Tools
15mThree key considerations for deploying best-of-breed observability
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
01h 02mCentralised Logging without the blood, sweat and tears
31mMy Logs Cost More Than My Services
36mScaling Log Management: PB-Scale Migration From Elasticsearch To...
46mConquering Distributed Cloud Logging Without Breaking the Bank