Data Prep Kit: A Comprehensive Cloud-Native Toolkit for Scalable Da...
summary
Open-source Data Prep Kit (DPK) assembles concrete components — a pdf2packet extractor, chunking, deduplication and embedding steps — and runs them at scale on Ray and Spark runtimes or pure Python; it ingests S3/NFS/local sources, stores columnar records with Apache Arrow/parquet, and automates pipelines via Kubeflow. Example use: cleaning 4,000 GitHub repos for HDL code, repeated acquire/tokenize/quality-analysis cycles, and it already powers IBM’s Granite models under an Apache-2.0 license.
AI-written summary · about
chapters (8)
- 0:00Data Prep Kit intro
- 1:29What is Data Prep Kit
- 9:13Data prep challenges
- 12:26How Data Prep Kit solves them
- 18:11How Data Prep Kit works
- 23:11Customizing transforms
- 26:25Live demo
- 35:17Future plans
related talks
40mPreparing data for LLM training with Data Prep Kit
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case Study
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case S...
26mOne Pipeline to Rule Them All: Unifying Multimodal and AI Data Processing with Daft
27mData Prep for Graphs
33mBridging Big Data and Machine Learning Ecosystems: A Cloud Native Approac...
23mDaft - The Distributed Python Dataframe for Complex Data
37mTaming the Data Tsunami