Log Rehydration and Logs to Metrics
summary
Datadog decouples ingest from indexing so teams ingest hundreds of terabytes a day while indexing a small sample (example uses 5%) and still archive everything to S3. A new JavaScript library automatically captures console logs, URLs, client IPs and user agents for front‑end correlation. Logs can be summarized into log‑based metrics at the index level, retained 15 months, and fed into anomaly and forecasting models. Cold archives are queryable via log rehydration that pulls just matching events (often <1 minute); Hulu example: 28M subscribers, 20 TB rehydrated, one scraper hitting ~8,300 errors in 30 minutes.
AI-written summary · about
chapters (6)
- 0:00Introduction to Datadog logs
- 2:06Front-end monitoring with JavaScript
- 3:52Generating metrics from logs
- 5:41Log rehydration solution
- 7:41Hulu case study
- 15:13Conclusion and Q&A
related talks
38mLog Analytics: Understanding Complex Systems at Scale
31mHow the Metrics Backend Works at Datadog
40mLarge-Scale Logging Made Easy
12mUnlocking the Secrets of Client Data
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
24mNew approach to metrics observability for the cloud era
40mObservability that scales: Analyze data in stream and query remotely
32mPowering cloud modernization w/log observability in context & at scale