Big Data and Small Data Everywhere, All at Once
summary
A large-scale data profiler gets ported to run entirely in the browser and it actually handles realistic loads: a client dataset of 6.5 billion records producing ~65 trillion value checks becomes explorable locally with no server and no data exfiltration. The implementation uses Spree (Scala profiler and rule execution engine) compiled via Scala.js → JavaScript, integrates Spark-style dataframe operations, supports interactive tables, histograms and rule aggregation, and the demo shows the approach working end to end.
AI-written summary · about
chapters (13)
- 0:00Porting big data to small environments
- 1:48Speaker background
- 3:54Run big data locally
- 5:20Data profiling and masking
- 7:07Data quality rules in browser
- 9:15Scala.js for big data in browser
- 14:17Spree data profiler
- 16:46Interactive data frame interfaces
- 21:03Spree features demo
- 24:42Future work and Unicode
- 26:06Spree API compatibility with Spark
- 27:52Real-time and WebAssembly improvements
- 29:18Try Spree and closing
related talks
53mTips and tricks for robust big data applications
38mWeb Development To Big Data: A Journey
01h 05mData Visualization with Scala tools
42mScala: The Unpredicted Lingua Franca for Data Science
29mKEYNOTE: Big Opportunities in Small Data
41mVisualizing big data
44mProcessing large data with Apache Spark
35mData Processing Pipelines for Small Big Data