The power of intuition in data science, and why it will always have a role
summary
Argues that intuition is a learned system view that lets people imagine counterfactuals and spot causal links data alone misses. Gives concrete methods: shadowing creative artists for two weeks before modeling, running a two‑week dataset experiment that shows forty percent of certain e‑commerce transactions depend on signals an engineer insisted on, and using A/B tests, randomized trials, supervised and deep learning plus Pearl’s causal hierarchy to formalize assumptions. Treats intuition as an internal simulator to generate hypotheses and then commits to testing them.
AI-written summary · about
chapters (9)
- 0:00Start
- 1:04What started us on this enquiry..
- 7:58Our 3 level hierarchy of intuition
- 10:37Intuitive judgement: sketching a definition
- 13:16Applying this as a Data Scientist - 3 tips
- 22:58Intuition as the system view
- 26:23What would the science seek to achieve?
- 28:01Aligns with 3-layer causal hierarchy (Pearl)
- 30:15So keep your job as a data scientist! 3 tips
related talks
26mThe Intelligence of Instinct
29m🧠🧹 Causality - Mental Hygiene for Data Science
45mCausality and Statistics, a Love Not So Impossible After All
01h 03mExperimentation in Data Science
41mA Practical-ish Introduction to Data Science
42mAugmenting Human Decision Making with Data Science
04mJustin Basilico interview
37mLife of a data scientist