Fast log management for your apps
summary
Shows concrete changes that cut log overhead: replace eager messages with a lazy logger with Supplier to avoid expensive computations, use async appenders with tuned buffer size and drop policy, and emit send JSON directly so parsing cost disappears. At Exoscale the pipeline becomes syslog-ng → Kafka → Elasticsearch, letting Kafka handle billing/stream processing and delaying indexing. Notes tradeoffs: durability versus speed, costly metadata (line numbers) and using hot-reloadable logging config to enable expensive fields only when needed.
AI-written summary · about
chapters (12)
- 0:00Introduction and background
- 1:09Problem with naive logging
- 4:33Lazy logger solution
- 6:00Physical storage impact
- 7:22Asynchronous logging trade-offs
- 10:19Centralized log management
- 12:04Structured logging benefits
- 13:50Direct JSON logging
- 15:32Disk scraping alternatives
- 16:38Hot reloading configuration
- 17:20XO Scale implementation
- 18:25Summary and best practices
related talks
39mFast log management for your infrastructure
48mLogging and ranting
31mThe Long and Winding Road to Application Logging [A]
31mLogging at Scale, Billion+ messages, 100K msg/sec
21mLogging at scale using Graylog - Billion+ messages, 100K req/sec
31mMy Logs Cost More Than My Services
46mBuilding a scalable logging platform
10mStructured Logging and You