You Can't Just Send All Logs to Elastic

on YouTube ↗53m

" summary

The talk explains why sending all application logs to Elasticsearch is not as straightforward as it seems, focusing on Elasticsearch’s characteristics for log storage, pitfalls of Java clients and other clients, and related engineering anti-patterns. It covers practical issues with using Elasticsearch for logs, high-load considerations, and best-practice recommendations.

AI-written summary · about

" chapters (24)

1. 0:00Introduction
2. 1:29Why not send all logs to Elastic
3. 3:35Elastic for logs vs text search
4. 6:04Index separation in Elastic
5. 7:47Making Elastic user-friendly
6. 9:12Log structure and field issues
7. 10:56Field limit in Elastic
8. 12:20Elastic REST client
9. 17:14Elastic version upgrade pitfalls
10. 20:52Client errors and failover
11. 23:23Apache HTTP client issues
12. 24:07Cluster topology updates
13. 28:24Data compression for Elastic
14. 29:28Apache HTTP client settings
15. 32:00Client settings summary
16. 33:02High Level client pros and cons
17. 34:28Bulk indexing in Elastic
18. 37:58Idempotency and unique IDs
19. 41:04Document indexing performance
20. 42:48Optimal bulk request size
21. 44:12Bulk request error handling
22. 45:37Elastic exception analysis
23. 47:24Talk summary
24. 49:09Q&A

" related talks

• 54m7 Petabytes of Logs in Elastic. How We Did ItRoman Nikolaev · Saint HighLoad++ 2023
• 42mMake your logs work for youHonza Kral · PyCon Australia 2015
• 31mFast logsNicolas Fränkel · JCON 2020
• 47mGraylog: one tool to log 'em allPiotr Przybył · JDD 2017
• 49mElasticsearch, Logstash & KibanaAlexander Reelsen · code.talks 2014
• 40mBuilding Observability with Elasticsearch: Practices for High LoadAnton Prykhodko · Highload fwdays'25
• 25mOne Size Does Not Fit All: A Polystore Architecture for Logs and TracesSuman Karumuri · Observability Summit North America 2026
• 54mScale Elasticsearch for Your Java Applications. Philipp Krenn. Java Fest 2020 OnlinePhilipp Krenn · Java Fest 2020