CG-Bench: Can Language Models Assist Call Graph Construction in the Real World?
summary
Benchmark contains 104 call sites drawn from seven large C/C++ projects (GCC, FFMPG, radius, ZFS, cure, open SSH, WRK) and isolates hard cases like dynamic linking via string concatenation and macro-driven indirection. Four open-source code LMs (CodeLlama variants, Deepseek) undergo zero‑shot, improved zero‑shot, and few‑shot prompts; all models exceed 50% accuracy and Deepseek-6.7b reaches 69.70% with few‑shot. Results show models exploit names, comments and string reasoning to resolve many indirect calls but still fail when irrelevant code or misleading identifiers dominate.
AI-written summary · about
chapters (7)
- 0:00CG-Bench project intro
- 1:27Graph construction challenges
- 2:39CG-Bench benchmark design
- 4:44Experimental setup & evaluation
- 6:54Key findings
- 8:24Future of program analysis
- 9:29Conclusion & availability
related talks
15mInterleaving Large Language Models for Compiler Testing
19mSecureMind: A Framework for Benchmarking Large Language Models in Memory Bug Detection(…)
18mReductive Analysis with Compiler-Guided Large Language Models for Input-Centric Code(…)
40mSurviving the Agentic Hype with Small Language Models
18mA Cocktail Approach to Practical Call Graph Construction
25mEvaluating An LLM's Ability To Code
33mPractical Evaluation of LLMs for Code
36mBeyond Grep: Search for Reliable Coding Agents