~/talk

Cheap DIY Monitoring with OSS Analytics & Visualization

on YouTube 30m
Talkat Conf42 DevSecOps 2023 · on Thursday, 2023-11-30· [observability, devops-and-delivery]

speaker

name:Robert Hodges
x:dbcicero

resources

video:https://youtu.be/ItF7-ksHpJM
slides:none on file

summary

Builds a VM stat collector in ~100 lines of code that pipes vmstat JSON into ClickHouse and visualizes it with Grafana so you can query raw time series with SQL, see CPU and memory traces live, and zoom into seconds. Demonstrates ingestion via curl/JSON each row, storage with MergeTree partitioned by day, compression often 90–95%, and live load tests using sysbench and stress that reveal collector gaps when the host is saturated. Plugin used has ~12M downloads and ClickHouse had 392 unique PR contributors last year.

AI-written summary · about

chapters (24)

  1. 0:00intro
  2. 1:43preamble
  3. 2:07about robert
  4. 3:21monitoring is for answering questions
  5. 3:55what's the best way to answer these questions?
  6. 4:36off-the-shelf solutions? perhaps not for you...
  7. 5:23let's build a monitoring system with open source
  8. 6:06pick an opne source analytic database
  9. 7:22a short list of reasons why clickhouse is popular
  10. 8:50clickhouse optimizes for fast response on large datasets
  11. 10:35and supports [many] dozens of input formats
  12. 11:16it also has great support for time-ordered data
  13. 12:30grafana pairs well with clickhouse for observability apps
  14. 14:00sooo... how do we ingest vmstat data and display it?
  15. 14:23step 1: generate vmstat data
  16. 15:12here's the output
  17. 15:38step 2: design a clickhouse table to hold data
  18. 17:28step 3: load data into clickhouse
  19. 18:23step 4: build a grafana dashboard to show results
  20. 19:26step 5: go crazy!
  21. 20:12demo
  22. 26:38more software to build monitoring on clickhouse
  23. 27:51where can i find out more?
  24. 28:54thank you and have fun!

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.