PyData NYC 2023
2023-11-01..2023-11-03
New York, United States
[PyData NYC 2018, PyData NYC 2022, PyData NYC 2023, PyData NYC 2024]
who it's for
data scientists, machine learning engineers, data engineers, analytics engineers, platform engineers, data visualization engineers
topics
Polars DataFrames in Rust; Apache Arrow with pandas and Dask; Quarto dashboards from Jupyter; GPU acceleration for scikit-learn; diffusion models for medical imaging; causal inference with dowhy and econml; Metaflow and Argo orchestration; LLMs in ETL and self-service analytics; retrieval augmented generation document apps; benchmarks: Spark Dask DuckDB Polars; interactive apps with Streamlit and Shiny; JupyterLab extensions and Codespaces integration
| title | author | length |
|---|---|---|
| Keynote: AI & the stuff built for AI - are they actually useful for data science? | Soumith Chintala | 32m |
Trendy tools get judged by concrete results: PyTorch adds numpy compatibility and a compiler that shows 57× CPU and 200× GPU toy speedups on microbenchmarks… | ||
| Keynote: Dashboards with Jupyter and Quarto | J.J. Allaire | 40m |
Announces Quarto Dashboards that render production‑quality dashboards directly from Jupyter Notebooks by converting notebook cells to dashboard “cards,” with… | ||
| Keynote: Scientific Discovery: From the Lab Bench to the GPU | Michelle Gill | 36m |
A production platform called BioNeMo combines a training framework (based on NVIDIA NeMo) and an inference service accelerated on GPUs. A small‑molecule… | ||
| How to Empower Utility Vegetation Management | Kewen Gu, Anjani Prasad Atluri | 30m |
LiDAR plus satellite imagery produce line‑level, foot‑level encroachment estimates aimed at one-foot precision, after satellite 30–50 cm imagery proves… | ||
| Polars; DataFrames in the multi-core era | Ritchie Vink | 44m |
Polars is an OLAP DataFrame engine written in Rust that targets modern hardware with cache‑aware data structures, lazy evaluation and a work‑stealing thread… | ||
| A Practical Guide to Causality in Python (For The Perplexed) | Aleksander Molak | 43m |
Practical steps and code in Python: builds causal graphs, encodes assumptions, uses front-door/back-door/do-calculus to translate queries into estimable… | ||
| The Billion-Request Content Recommendation System Challenge | Dhasmana et al. | 34m |
Claim: a recommendation API that scales to ~1.3 billion pageviews per day and shifts heavy work to background to cut median latency from ~2s to milliseconds… | ||
| Deciphering Sales Drivers at PepsiCo | Suri Chen | 29m |
Compares Bayesian and Frequentist regressions applied to Media Mix Modeling, using lightweightMMM JAX NumPyro to fit nonlinear ad-response curves, priors… | ||
| Synthetic Image Generation in Rare Disease Diagnosis | Dobrindt et al. | 29m |
Claims that synthetic chest x‑rays improve rare‑disease classifiers, demonstrated by two demos: an image‑to‑image pipeline using a latent diffusion model… | ||
| Crafting Reliable Rules with Robust Control | Michael Zargham | 28m |
Applies Robust Control to tune a pricing algorithm for a collateralized lending platform, shares code and design digital twin repos and uses the open source… | ||
| Building Robust Reactive ML Systems In A Multiplayer Setting | Valay Dave | 35m |
Claim: Metaflow provides an event driven pipeline that version controls artifacts, freezes Conda/Pip deps, and auto launches steps on Kubernetes then schedules… | ||
| Bad data - anecdotes and examples from the real world | Avishek Panigrahi | 35m |
Catalog of real failures and fixes: concrete examples of corrupted CSVs, mixed units, locale date swaps that turn March into May, and typos that split… | ||
| The Merger of Machine Learning, Music and Programming | Ramon Perez | 51m |
Claims concrete results: unconditional and conditional audio samples generated with MusicGen and MusicLM style models, vocal cloning that convincingly imitates… | ||
| A practical guide to analysis and interactive visualization of massive datasets | Kim Pevey | 01h 25m |
Hands‑on setup of an opinionated Nabari JupyterHub on GCP provisions per‑user servers, then uses Dask (via Dask Gateway) to read ~70 GB (CSV→parquet) flight… | ||
| The Arrow revolution in pandas and Dask | Patrick Hoefler | 31m |
Pandas and Dask adopt Apache Arrow so string columns become contiguous and memory efficient and parquet reads can be zero‑copy leading to 50–60% memory… | ||
| Open Source Computational Economics: The State of the Art | Sebastian Benthall | 32m |
Open source toolkits like diner, Doo and Quant Eon provide model DSLs and solvers but struggle with high dimensional heterogeneous agent models so recent… | ||
| Harnessing Test-Driven Development & CI/CD for Smarter Data Analysis | Liz Johnson | 35m |
Builds a movie recommender from a 40,000‑row Kaggle set using TF‑IDF and cosine similarity, implements pytest unit tests with fixtures and parametrize (tests… | ||
| Basics of cloud computing for data scientists | Gastón Barbero | 24m |
Claims that basic cloud deployment can be practical for data scientists by showing concrete AWS patterns and trade offs: compute choices (EC2 vs serverless… | ||
| Modern Data Pipelines Testing Techniques: A Visual Guide | Moussa Taifi | 39m |
Claim: disciplined testing turns fragile scripts into reliable pipelines by shortening feedback loops to under five minutes when possible. Shows an end-to-end… | ||
| How to Use Python to Better Understand the Impact of Electricity Pricing on Consumption | Saba Nejad | 40m |
Processes the Low Carbon London trial with Python and pandas at half hour resolution across 167 million readings and builds a counterfactual estimator by… | ||
| Scikit-learn on GPUs with Array API | Thomas J Fan | 36m |
Scikit-learn 1.3 runs a subset of estimators on accelerators by dispatching array operations to CuPy and PyTorch via the Array API standard, yielding… | ||
| The Beauty and the Beast: Python on GPUS | Andy Terrel | 36m |
Python code runs correctly on GPUs only when it handles memory and threading explicitly or uses tooling that does it for you: low‑level CUDA and CUDA Python… | ||
| Spark, Dask, DuckDB, and Polars: Benchmarks | Matthew Rocklin | 37m |
Benchmarks run Spark, Dask, DuckDB and Polars on scales from 10 GiB to 10 TiB using TPC-H on a local M1 MacBook Pro and AWS instances, with orchestration via… | ||
| Building an Expert Q&A Bot with Open Source Tools and LLMs | Chris Hoge | 01h 00m |
Claims building an expert QA bot that answers questions about an open‑source project by pulling GitHub docs into a vector DB, using LangChain for loaders and… | ||
| Half hour of labeling power: Can we beat GPT? | Ines Montani, Ryan Wesslen | 45m |
Claims that LLMs are best used at development time to pre‑annotate and speed dataset creation, then fine‑tune smaller task‑specific models to match or beat… | ||
| Adding Your Own Data Apps to JupyterLab | Daniel Goldfarb | 49m |
Step-by-step guide shows how to build a real front-end JupyterLab extension from the JupyterLab extension template, using TypeScript, Lumino widgets and JPM… | ||
| Ibis: A fast, flexible, and portable tool for data analytics | Phillip Cloud, Gil Forsyth | 01h 17m |
Ibis gives a pandas‑like API that builds deferred execution expressions and compiles them to dialect-specific SQL for backends like BigQuery Snowflake Spark… | ||
| Using Open Source LLM in ETL | Mei Chen | 29m |
Case study uses Llama2-70b to convert applicant Q&A into features for reinsurance ETL by doing binary disease classification + chaining + regex→JSON, evaluates… | ||
| Solving the problems in front of you | Randy Au | 26m |
Argues for finding the minimal right amount of upfront design and shows concrete practices: start with a working prototype against real inputs (nightly CSVs… | ||
| Self-Service Analytics using LLMs | Aaditya Bhat | 39m |
A concrete recipe for a production-ready LLM-based self-serve analytics platform: connect a data catalog connector, use an LLM as query generator to emit SQL… | ||
| Machine Learning in your Data Warehouse using Python | Megan Lieu | 21m |
Loads telecom rows into Snowflake via Snowpark for Python, transforms into four dimension tables with the DataFrame API, trains a random forest in Deepnote and… | ||
| Low(er) Code ML Pipelines with Conduit | Viren Bajaj, Piero Ferrante | 22m |
Conduit is a Python package plus a terminal UI that claims to cut time-to-production by templating repos, letting teams paste notebook code into templates with… | ||
| Leveraging Generative AI for Enhanced E-commerce | Kriti Kohli | 37m |
Shopify reports deployed features: AI-generated product descriptions, email copy and scheduling, and storefront chat instant answers serving millions of… | ||
| From RAGs to riches: Build an AI document interrogation app in 30 mins | Eswaramoorthy, Pothina | 41m |
A working RAG pipeline gets built end to end: Ragna OSS framework wires embeddings, vector stores and LLMs; a Panel web app provides the chat UI; models… | ||
| Forecasting With Classical and Machine Learning Methods | Jonathan Bechtel | 39m |
Claim: machine learning can beat classical forecasting but only with careful preprocessing and feature design. The demo uses SKTime to run auto ARIMA… | ||
| Architecting Data: A Deep Dive Into the world of Synthetic Data | Ramon Perez | 36m |
Claims that synthetic data lets teams replace sensitive inputs and build a full‑stack product: five interrelated projects are produced — reproducible data… | ||
| Using Generative AI and Foundation Models to Predict Above Ground Biomass | Singh, Srinivasan | 42m |
Claims a scalable method to estimate above ground biomass density worldwide by fine‑tuning a geospatial foundation model on multispectral satellite imagery and… | ||
| Turning your Data/AI algorithms into full web applications in no time with Taipy | Florian Jacta | 34m |
Taipy is an open-source Python library that claims to turn algorithms into full apps with both front-end and back-end controls — demoed as 17 lines of code to… | ||
| Ten Years of Community Organizer | Alexander CS Hendorf | 43m |
Ten years of organizing work claim concrete wins: program chair for 5 years running selection and recordings, migration from ad hoc Google Drives to documented… | ||
| Taming the toxic python environment on your laptop | Dharhas Pothina | 41m |
A desktop+cloud UI called conosur (cond store in the transcript) actually builds reproducible artifacts: it creates a lock file with exact hashes, produces… | ||
| Simplifying Data Analysis with GitHub Codespaces, Jupyter Notebooks & Open AI | Nitya Narasimhan | 42m |
Shows a concrete workflow: GitHub Codespaces + a reproducible dev container boots a ready Python Jupyter environment, GitHub Copilot creates and iterates a… | ||
| Enabling continuous deployment with stealth, security, & smooth moves in mind | Erin Mikail Staples | 56m |
Claims a hands‑on path: architecture notes for scalable databases, a demo building FastAPI backend and Next.js front end, and using the LaunchDarkly feature… | ||
| Sciris: Simplifying scientific software in Python | Cliff Kerr | 34m |
Sciris claims to cut boilerplate and make common scientific tasks explicit by wrapping NumPy, pandas, Matplotlib and low‑level serializers (pickle/dill) and by… | ||
| Plotting with Matplotlib; Telling Static, Animated, & Interactive Stories | Hannah Aizenman | 59m |
90-minute tutorial that teaches the Matplotlib object model by building real outputs: exploratory charts from the Palmer Penguins dataset, a linked animation… | ||
| Opening Remarks: Martha Norrick | Martha Norrick | 11m |
Welcome remarks outline concrete civic work: the Office of Data Analytics (established 2012) uses Python and R and open-source tooling to run predictive models… | ||
| Turning notebook into a web app with Mercury | Płońska, Płoński | 47m |
Claims that MERCURY turns a Jupyter Notebook into a working web app, report, dashboard and interactive slideshow with no frontend code; demo uses pandas… | ||
| Lightweight, low-overhead, high-performance: machine learning directly in C++ | Ryan Curtin | 43m |
Claim: rewriting pipelines in C++ yields much smaller and faster deployments. Uses mlpack + Armadillo with low-level BLAS (OpenBLAS) and aggressive statically… | ||
| Lightning Talks Day 2 | Erin Mikail Staples, Jessica Garson | 29m |
Short, concrete demos and tools dominate tonight: an open source wrapper called lfs spec that plugs data versioning into FS spec reads and writes and adds… | ||
| How to Become a Successful Tech YouTuber? | Dhaval Patel | 40m |
Claims a concrete 5-step framework and shows metrics: 55,000 views per day, a 38,000-person Discord community, and over $500,000 in course revenue in one year… | ||
| Dask tutorial | Patrick Hoefler | 01h 26m |
Shows Dask (local and distributed) used to parallelize Python: turning slow for‑loops into futures‑based parallel tasks, scaling pandas workflows to a 100 GB… | ||
| Comics in NumPy? More Likely than You Think! | Mars Lee | 31m |
A completed 16‑page NumPy Contributor Comics project funded by Google Season of Docs claims comics can carry concrete docs: hand‑drawn pages (explicitly no… | ||
| Building an Interactive Network Graph to Understand Communities | Lucas Durand | 01h 26m |
Claim: building an interactive network graph that replaces an org chart by clustering people on real attributes and finding proximity. Implementation uses… | ||
| Build Simpler Production ML Systems using Feature/Training/Inference Pipelines | Fabio Buso | 01h 12m |
Claims a unified Feature Training Inference pipeline pattern and shows a working loan‑approval demo built on an open feature store and model registry: feature… | ||
| Build Simple and Scalable Apps with Shiny | Gordon Shotwell, Tracy Teal | 35m |
Claims that Shiny for Python infers a reactive computation graph and builds it by runtime tracing so only affected outputs re-render. Shows concrete… | ||
| Creating Interactive, Animated Reports in Streamlit with Vizzu | Blackwood, Vidos | 58m |
Integration of ipyvizzu with Streamlit actually ships a C++ morphing engine compiled to WebAssembly, exposes animated chart morphs and exportable HTML stories… | ||
| Simple Simulators with pandas and Generator Coroutines | James Powell | 01h 22m |
Builds a backtester that actually runs in Python using plain pandas Series inputs/outputs and generator coroutines to model strategy and simulator loops… | ||
| Faster SQL with pandas and Apache Arrow | Will Ayd | 41m |
Pandas connects to databases using new Apache ADBC drivers that pass Arrow columnar data and move work into compiled code so SQL I/O gets both correct types… | ||
| Covalent: A New Paradigm for High Compute Cloud Workloads | Kumar Radha, Ghukasyan | 01h 26m |
Covalent claims a Python-first orchestration layer that actually ships functions to real compute: decorator-based decorator-based electrons turn Python… | ||
| Innovation in the Age of Regulation: Federated Learning with Flower | Krishi Sharma | 32m |
Demonstrates that training without centralizing sensitive records is practical: a demo uses Flower to run a server on port 880, spin up clients running simple… | ||
| Adventures in not writing tests | Andy Fundinger | 31m |
Claim: type hints plus Hypothesis generate meaningful tests so you spend less time handcrafting examples. Tools used: Hypothesis Ghostwriter to scaffold tests… | ||
| nothing matches | ||