Centralised Logging without the blood, sweat and tears
summary
Shows a working pipeline that replaces a 171‑server, $38k/month legacy stack with a managed AWS setup using Kinesis shards → Lambda consumer → Amazon Elasticsearch (2.3) + Kibana deployed via Terraform. Claims one node + 5 Kinesis shards can handle ~1M messages/day for roughly $145/month (vs billions/day scale previously), demonstrates live green cluster and indexed data, and publishes the Terraform + Lambda code to GitHub so the example is reproducible.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 2:04The problem with logs
- 7:01Evolving logging architectures
- 17:15Modern logging with AWS
- 21:45Deploying with Terraform
- 31:40Configuring AWS components
- 39:11Elasticsearch and visualization
- 51:07Cost analysis and conclusion
related talks
32mBuilding and scaling a log analytics platform - a serverless approach
39mServerless in production, an experience report
01h 02mServerless in production, an experience report
50mTo Normalized Logs, and Beyond - Building a Threat Detection...
42mServerless Tales from the Trenches
46mConquering Distributed Cloud Logging Without Breaking the Bank
46mBuilding a scalable logging platform
40mLarge-Scale Logging Made Easy