State of the art natural language understanding at scale
summary
Open-source Spark NLP provides production-ready, enterprise-grade NLP with about 90k downloads/week and ~1M downloads per quarter, 26 releases in 2020, and over 300+ pre-trained models. Claims include state-of-the-art NER (≈94% on CoNLL‑2003), language detection at 97–99% for full texts, a deep‑learning context spell‑checker with ~8% word error on a benchmark, CPU/GPU‑optimized builds, and live Python notebooks and Streamlit demos that run and scale on Spark clusters.
AI-written summary · about
chapters (8)
- 0:00Start
- 1:46Introducing Spark NLP
- 3:45The most widely used NLP library in the enterprise
- 6:35Spark NLP: Supported Environments
- 10:41Using Named Entity Recognition
- 13:21Accuracy: State-of-the-art Models Multi-class & Multi-label Text Classifications
- 15:43Using Sentiment Analysis
- 25:37Scale: Distribution & Parallelism
related talks
30mBuilding Complete State of the art Natural Language Processing
30mTurbocharging State-of-the-art Natural Language Processing on Ray
50mDeep Learning and NLP with Spark
45mDeep Learning and NLP with Spark
12mUnlocking the Power of NLP: Techniques in K8s-Native Applications
38mNatural Language Understanding at Scale
55mDemystifying Natural Language Processing across several industry verticals
39mMachine Learning with Scala on Spark