QUARK: A PURELY-FUNCTIONAL SCALA DSL FOR DATA PROCESSING & ANALYTICS
summary
Quark is a purely-functional Scala DSL that runs on the Quasar Analytics compiler and processes semi-structured data, compiling query plans into operations that run entirely inside target backends (MongoDB, Couchbase, MarkLogic, Hadoop/HDFS). It uses a finally tagless design to produce type-safe descriptions instead of serializing opaque Scala closures, so plans can be optimized and pushed down to the datastore. The library is open source (github.com/quasar-analytics/quark) and pairs with quasar-analytics/quasar as the compiler backend.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 1:00Apache Spark
- 4:22Why Does Spark Suck
- 5:18What Would Functional Programming Do
- 6:50The DoNothing Rule
- 8:21What is Quark
- 10:34Quark vs Sparky
- 11:00QUARK Targets
- 12:11Building a Scala DSL
- 13:58Extending a Scala DSL
- 17:03Bugs
- 17:32Finally Tagless
- 21:28Building Blocks
- 23:42MiniPork
- 23:52Type System
- 24:34Set Level Operations
- 27:05Mapping Operations
- 28:30Awkward Programming
- 29:33Mapping Func
- 30:26Mapping Function Arrows
- 31:52Outro
related talks
23mSpark Datasets: why they aren't great and what could be done
56mEfficient array (and data) processing by leveraging structure
25mIntroducing Quine: a Streaming Graph for Modern Data Pipelines
30mReconciling Kafka, SQL and Parquet
23mQuery like a Pro: ES|QL meets Apache Arrow and Pandas
32mStreaming Graph Search with Quine.
25mA quack of all trades: DuckDB, a versatile analytical database to keep in your toolkit
37mCascalog 2- Streaming MapReduce in Clojure