Inequality Joins in Pandas with Pyjanitor
summary
pyjanitor adds a conditional_join to pandas that implements inequality joins via sorting and binary search so the code actually reduces search space instead of doing a Cartesian product. Two implementations exist: a Python non-number version (easy to read but slow and memory-hungry) and a faster number implementation that parallelizes. Examples: 30k-event overlap case drops from ~900M Cartesian comparisons to milliseconds (non-number ~700 ms, number ~17 ms); large tests show seconds vs minutes. Future work: integrate into pandas/pidata table.
AI-written summary · about
chapters (9)
- 0:00Inequality joins in Pandas
- 3:21What joins and inequality
- 6:29Pyjanitor's conditional join
- 9:19How conditional join works
- 15:26Non-numpy implementation
- 15:46Numpy range join algorithm
- 19:21Region-based range joins
- 21:55Performance benchmarks
- 25:29Future work and integration
related talks
39mMastering DataFrame Diffing Techniques
43mWhy do I need to know Python- I'm a pandas user
28mIdiomatic Pandas
35mIdiomatic Pandas
40mBeyond Pandas: The Great Python Dataframe Showdown
32mPandas May Be Slow, but Pandas Doesn't Have to Be
01h 17mPandas from the Inside / "Big Pandas"
01h 21mPandas from the Inside