What's with the 1s and 0s? Making sense of binary data at scale
summary
Uses Apache Tika to auto-detect file types, extract metadata and plaintext, and chain Tesseract OCR for images; the distribution ships ~70 format parsers and exposes tika-server and tika-app plus a tika-eval batch runner for large corpora. Detection combines filenames, magic bytes, n‑gram language/encoding models and container unpacking; at scale (think millions to 10 million files) expect misidentifications, PDFBox extraction bugs that corrupt text, OOMs and hangs, and the practical fix of isolating parsing in a separate JVM or server pool.
AI-written summary · about
chapters (24)
- 0:00Intro
- 1:20Detecting files
- 5:35Container formats
- 6:44Making sense of containers
- 9:13Machine learning
- 13:00Engrams
- 14:25Text vs binary
- 16:17File formats
- 17:35Calling tikka
- 18:56tikka app
- 20:13tikka server
- 21:30Tikka Facade
- 23:13forked passer
- 24:18extending tika
- 25:47plan for failure
- 27:30bug reports
- 29:33tika breaking your JVM
- 30:10PDF file corruption
- 31:05CV file corruption
- 32:04What will go wrong
- 36:00TKeval
- 38:00TKeval Batch
- 39:10Other Apache projects
- 40:03Other nonapache projects
related talks
36mTika 2.0 -- Robustness and Scale
36mScalably crashing JVMs, or why binary data to content is hard
48mIndexing Your Office Documents
39mAdventures in Understanding Documents
45mSafely Capture Institutional Knowledge Using
44mAutomatically scale a serverless app with Amazon Textract & MongoDB
18mData Classification at Scale: Taming the Hydra
47mYou Got 10 Million PDFs from the Ministry. Now What?