Move beyond academia: Introducing an industry-first tabular benchmark
summary
A benchmark and a pipeline package that run complex pre/postprocessing on real industry problems and synthetic proxies, built from private industry datasets plus public ones and exposed via a results dashboard. It reports that the first release has 50 datasets now, 200 planned and 36 industrial tasks growing to 65, evaluates models (CARTE, TabICL, TabPFN, gradient boosting, ensembles, MLPs) on metrics like Precision@K/Recall@K/MAP@K, and finds foundation models outperform gradient boosting and MLPs with surprising disagreement versus existing benchmarks.
AI-written summary · about
chapters (11)
- 0:00Tabular benchmark project
- 1:27Existing tabular benchmarks' flaws
- 3:59Benchmark result discrepancies
- 5:24Industry challenges in tabular ML
- 7:52Sharing pipelines securely
- 10:00Tabbench features
- 12:11Benchmark results and comparisons
- 14:39Insights from industrial datasets
- 16:08Synthetic data for academia-industry gap
- 19:41Conclusion and future directions
- 21:50Q&A
related talks
36mRun a benchmark they said. It will be fun they said.
25mHumans helping machines help humans run machines
44mKeynote: The neglected art of a relevant benchmark and why you need one
01h 28mHands-On with Tabular Foundation Models
49mBuilding Advance Analytics pipelines with Azure Databricks
13mPremiere
25mTorchBench: Guarding the Performance of the PyTorch Ecosystem with Continuous Benchmarking
44mDemystifying Benchmarks: How to Use Them To Better Evaluate Databases