Revolutionizing ETL for RAG with
summary
Sycamore is an open-source ETL system built on Ray that turns messy PDFs into a tree-structured Sycamore DocSet for scalable ETL and analytics, using a custom partitioner (Formidable DETR trained on DoLayet) that shows 2–4× segmentation accuracy and 350k Hugging Face downloads. Pipelines run locally or on clusters, processed 100k+ documents overnight, lifted a customer support RAG baseline from 55% to 97%, materializes checkpoints to S3, and writes embeddings to OpenSearch; tables via vision LLMs remain a work in progress.
AI-written summary · about
chapters (19)
- 0:00Introduction and motivation
- 1:05Customer use cases for unstructured data
- 2:31Challenges with traditional RAG systems
- 4:14Importance of document ETL for AI applications
- 4:55Introducing Sycamore platform
- 6:00Segmentation problem in document processing
- 8:47Vision models for document segmentation
- 10:11Why Ray for document processing
- 12:17ETL with Sycamore transforms
- 13:41Ray Data vs Sycamore data model
- 15:49Document tree structure in Sycamore
- 17:11Semantic transforms for document analysis
- 17:54Sycamore script example with NTSB data
- 20:01Materialization and checkpointing in Sycamore
- 21:26Ray implementation details
- 22:29Challenges scaling Ray down
- 24:15Analytics queries beyond RAG
- 25:40Conclusion and future directions
- 26:42Q&A session
related talks
43mBeyond RAG - Going From Search to Analytics on Unstructured Data with Aryn
27mWhy Your RAG is Failing , And How Open Source Can Fix It
01h 28mFrom RAG to Riches Building Enterprise Search Systems with LLMs
23mNo RAGrets
36mDon't tell me RAG is easy
46mData Engineering for Graph-based Retrieval Augmented Generation GraphRAG using Neo4j
29mRAG for Data Engineers: Why Your Metadata Is the Missing Piece
33mRAG Pipelines Letting You Down? Fitch Group's Solution for Financial Documents