Node js applications diagnostics under the hood

on YouTube ↗25m

" summary

Shows a concrete production workflow: create core dumps selectively with gencore to fork and dump without killing the main process, extract stack and request data with llnode and node-report and match X-Transaction-ID from logs, and find memory holders with V8 GC tracing plus heap/core inspection. Uses 0x flame graph to pin a 65% server‑rendering hotspot down to 15% and reports overall performance improvement of about 50%, plus tactics for Event Loop lag, GC stalls and Async Hooks tracing.

AI-written summary · about

" chapters (30)

1. 0:00Intro
2. 0:32Customers in retail domain
3. 0:52Stages of Node.js diagnostics evolution before 2017
4. 2:24The third stage of Node.js diagnostics evolution
5. 3:01V8 engine update
6. 3:40Node.js in Enterprise Architecture
7. 4:18Debugging in production
8. 4:44Example
9. 5:29Core dump creation algorithm for postmortem debugging
10. 5:43Debugging steps
11. 6:04Get the StackTrace and find the last JS function
12. 6:45Read the source code of the function
13. 6:55Get input parameter - request
14. 7:29Case 1. Informative Stack Trace
15. 7:47Case 2. Unhandled rejection
16. 8:19Case 2. Uninformative StackTrace
17. 8:40Case 2. Finding all reg object instances
18. 8:57Case 2. Which request crashed the process
19. 9:07Case 2. List of the killed requests in current process
20. 9:37Case 2. Search request by address of headers object
21. 10:06Search for local variables
22. 11:22Errors in production
23. 12:37Error handler with Error Registry and gencore
24. 13:17Core dump creation algorithm in production
25. 15:12Profiling node --profiler + apache bench/JMeter
26. 16:26Data processing
27. 17:03Performance improvements
28. 18:18Memory grows
29. 18:33Case: Excessive garbage collection, The creation a lot of new objects
30. 19:30Case: Scavenge GC in New Space.

" related talks

• 57mTroubleshooting Node.js Applications Under the HoodNikolay Matvienko · HolyJS 2017
• 28mPost mortem NodeJS debuggingThomas Watson · JSNation 2019
• 50mDebugging Node.js in ProductionThomas Watson · HolyJS 2016
• 50mPost-Mortem Debugging in Node.jsThomas Watson · JS Fest 2019 Spring
• 38mWelcome to Mordor! Hunting Performance Issues in Node.js.Daniel Khan · Codemotion Amsterdam 2016
• 23mThrow Me a Lifebuoy: Debugging Node.js in Production with Diagnostic ReportsChristopher Hiller · Node+JS Interactive 2019
• 16mNode.js Diagnostic Best PracticesGireesh Punathil, Mary Marchini · OpenJS World 2021
• 28mDiagnostic Tooling - Be Ready when Things Go Wrong in ProductionMichael Dawson, Mike Kaufman · NodeConf EU 2018