Turbocharging State-of-the-art Natural Language Processing on Ray
summary
A Python library named nlu wraps Spark NLP so users run production-grade models with a one-line API that returns whichever dataframe type is passed (pandas, Modin/Ray, Spark). It ships 250+ pre-trained models (46 languages, embeddings, NER, sentiment, sarcasm, fake-news, spam) and leverages Modin on Ray to scale without API changes. Benchmarks show Intel CPU training beating an older Tesla P100 by ~90% in one case; examples run end-to-end and produce corrected tokens, entities, and embeddings.
AI-written summary · about
chapters (8)
- 0:00Introduction to talk
- 0:23Spark NLP overview
- 8:51NLU library introduction
- 19:10Integrating Ray with NLU
- 21:18Modin on Ray demo
- 24:06Named entity recognition demo
- 27:13Spell checking on Ray demo
- 28:57Q&A and resources
related talks
44mPanel: ML Platform on Ray
30mBuilding Complete State of the art Natural Language Processing
22mRay, a Unified Distributed Framework for the Modern AI Stack
17mScaling and Unifying SciKit Learn and Spark Pipelines using Ray
33mState of the art natural language understanding at scale
26mAirbnb's LLM Evolution: Fine-Tuning with Ray
24mRay: A System for High-performance, Distributed ML Applications
35mOperating a Production-Scale Physical AI Data Platform with Ray