Scalably crashing JVMs, or why binary data to content is hard
summary
Finding what a blob of bytes actually is breaks at scale: retries and speculative re-execution can spin a single bad PDF into a crashed cluster, container formats hide inner types, and misidentifying encodings yields garbage. Uses Apache Tika to combine magic bytes, extensions and n‑gram encoding heuristics, offers a forked parser (child JVM) and a batch mode for isolation, and runs a 3.5 terabyte regression corpus to measure regressions — but JVM quirks like a Java 8/9 infinite loop bug still take nodes down.
AI-written summary · about
chapters (34)
- 0:00Intro
- 1:35Disclaimer
- 2:23Retries
- 3:40Vehicle tracking
- 4:55Marking failures
- 5:21Detecting file types
- 6:20File extensions
- 6:44My magic
- 8:07Container formats
- 9:08Tika bug
- 9:49Cloudera help
- 10:58Encoding
- 11:48Patterns
- 12:12Engrams
- 12:34Text Length
- 13:30JVM Bugs
- 14:40Teeka infinite loop
- 15:57Java libraries external libraries
- 16:13Scientific file formats
- 16:37Metadata
- 17:05Failure
- 18:45Stack traces
- 19:05Test files
- 19:46Zip files
- 20:01Log
- 20:40Unit Test
- 24:33Memory leaks
- 25:23Regression corpus
- 26:45Reprocessing
- 28:10JVM crash
- 29:58forked parser
- 32:43Quickstart
- 34:13What are you going to do
- 34:45When you get failures
related talks
40mWhat's with the 1s and 0s? Making sense of binary data at scale
44mAtomic Habits in writing high performing distributed code in the cloud
36mTika 2.0 -- Robustness and Scale
44mMY AWFUL EX-DATASYSTEMS
34mBig Data, Small Code ... #bbuzz
32mCodebase Research
42mCatch me if you Can Processing Terabytes in Secs
39mAdventures in Understanding Documents