Mojibake! What the h—ck happened to these strings?
summary
A Python module called ftfy detects and undoes real-world mojibake — including piled-up cases that require decoding three layers to recover the original text — by applying hand-tuned heuristics and a big regexy “badness” metric rather than ML. It identifies common failure modes (Excel CSV exports, Windows 1252 misdecodes, broken heuristics like chardet), restores emojis and punctuation reliably, and claims false positives on the order of once in several gigabytes.
AI-written summary · about
chapters (9)
- 0:00Mojibake and ftfy intro
- 1:03ASCII and extended encodings
- 3:53Unicode and UTF-8
- 5:39How UTF-8 mojibake happens
- 7:26Fixing mojibake with ftfy
- 8:30Detecting mojibake patterns
- 9:39Root causes of mojibake
- 10:42Excel and mojibake
- 11:24Preventing mojibake
related talks
25mDecoding the Frustrations and Vulnerabilities of Modern Text Processing
24mHow I Used Python to Stop Worrying & Love Emoji in Bioinformatics
25mWhy len 4 and other weird things you should know about strings in Python
43mFun With Python and Emoji
35mText Is Not Text: The History of Strings in Python
49mWhy You Should Do Text Analysis in Python (Even if You Don't Want to)
28mIntroducing t-strings: f-strings with superpowers
28mMaking Acquaintances with Exotic Encodings