SQL layer over user listening history for agentic access
summary
Stateless Kubernetes Java gRPC pods ingest userID plus SQL, fetch Protobuf listening logs, convert to Arrow vectors and run queries in-process with DuckDB in-process, returning Arrow/JSON; median query responses are tens of megabytes with tail queries approaching a gigabyte, forcing heavy memory tuning and buffer reuse. Joins to catalog metadata use an existing distributed join service. The deployment supports agentic natural-language queries and ad-hoc data scientist experiments and jemalloc reduced memory by 40%.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 2:28User behavior data platform
- 3:57Flexible query interface for agents
- 4:59DuckDB integration
- 6:24Query patterns and memory
- 8:53Joins and data enrichment
- 9:57Democratizing user behavior data
- 10:39DuckDB Java with jemalloc
related talks
30mOLAP on Your Cassandra Data with Arrow, Flight SQL, ADBC, and DuckDB
40mDuckDB: Harnessing in-process analytics for data science and beyond
07mKalDB: Log analytics with DuckDB
26mBuilding Data Lake Using DuckDB
30mSparkless Patterns: The New Data Stack and Modern Data Architecture
40mDuckDB beyond the notebook
36mEmbrace the chaos: Composable data systems with fewer asterisks
34mData Wrangling with DuckDB