Moving Faster: Why Intent Media Chose Cascalog
summary
Company migrates from Java MapReduce, Pig and Vertica to Cascalog on Clojure/Cascading to speed model development and deployment, with an eight‑to‑nine person data team producing models that perform 5–10× better than larger competitors. Workflows run in a REPL that spins up a local Hadoop for fast unit tests and checkpoints; a monthly spike once consumed 5.3 years of CPU time in one month on EMR. Cascalog 2.0 and Cascading 3.0 target Spark/Storm backends to cut long MapReduce jobs toward near‑interactive runtimes.
AI-written summary · about
chapters (10)
- 0:00Introduction and context
- 3:24Evolution of data processing
- 6:48Moving to Apache Pig
- 9:26Cascading and abstractions
- 10:57Adopting Clojure and Cascalog
- 14:34The power of declarative logic
- 17:25Word count across frameworks
- 21:40Cascalog demo and workflow
- 26:05Workflow optimizations
- 31:27The future of data processing
related talks
41mIntroducing Cascalog: Functional Data Processing for Hadoop
37mCascalog 2- Streaming MapReduce in Clojure
01h 12mIntroduction to Cascading
46mCascalog: Making Data Processing Fun Again
39mComplex Data Transformations Made Easy
45mModern Data Architecture
33mUsing Kaskada to do Real Time Machine Learning
32mMove to managed analytics