Using Clojure+Spark to Find All the Topics on the Interwebs
summary
Uses Clojure+Spark and Spark MLlib to classify and topic-model web data, pulling seeds from DMOZ and pages from Common Crawl to expand to about 2.5 million pages for fashion and millions more for negatives. Builds bag-of-words filters, trains random forests for classification (F measure ~0.86 on raw features) and runs LDA topic models after reducing vocabulary. Production pipeline runs but LDA requires heavy tuning and memory workarounds.
AI-written summary · about
chapters (6)
- 0:00Start
- 8:34Clojure using Reducers
- 9:00Clojure using Spark
- 14:57Creating a single column DataFrame
- 23:14Computing metrics
- 26:17Computing Chi-Square Test
related talks
38mMachine Learning with Clojure and Apache Spark
28mLarge Scale Graph Mining with Spark: What I learned from mapping 15 million websites
44mData Science in Clojure
21mCase Law and ML on Spark
40mSmarter Search with Spark, Solr and Machine Learning
45mDeep Learning and NLP with Spark
29mThinking of Topic Modeling as Search
39mLive-Hack: Analyzing 7 years of Buzzwords at Scale #bbuzz