Choose, Don’t Label: Multiple-Choice Query Synthesis for Program Disambiguation
summary
A new active-learning system called Socrates generates multiple-choice queries (each option a Hoare triple) instead of asking users to label concrete inputs, synthesizes scenarios that distinguish candidate programs, and produces simple postconditions as answer choices. It evaluates on 157 tasks across four domains (tables, JSON, image edit, image search), solves 100% of benchmarks while baselines fail on 14–37%, yields 89% user answer accuracy versus 64% for input–output queries, and matches baselines on runtime aside from LLM natural‑language rendering overhead.
AI-written summary · about
chapters (13)
- 0:00Program disambiguation intro
- 3:50Key contributions
- 4:32Formal query definition
- 6:19Query synthesis steps
- 7:23Distinguishing constraints
- 9:34Answer choice construction
- 10:37Evaluation across domains
- 12:47Baseline setup
- 13:29Accuracy results
- 13:50User study on queries
- 14:34Runtime efficiency
- 15:16Motivation and trust
- 15:36Conclusion and Q&A
related talks
57mVariant analysis
21mThe Exploratory Labeling Assistant: Mixed-Initiative Label Curation with Large Document Collections
10mData-centric disambiguation for data transformation with programming-by-example
10mSporq: An Interactive Environment for Exploring Code Using Query-by-Example
41mCreating your own query language as an alternative to your forms
18mReality is not an end-to-end prediction problem
17mAccelerating Syntax-Guided Program Synthesis by Optimizing Domain-Specific Languages
29mAnnotating data the right way