Protecting data scientists in healthcare with type safety

on YouTube ↗22m

" summary

Wellframe moves ingestion out of ad hoc Python into a Scala pipeline that puts Kafka between sources and consumers and emits Scala case classes for strong QA the team stores schematized records in MySQL Cassandra and Elasticsearch then exposes data to analysts as PySpark DataFrames while migrating from RDDs to Spark Datasets to regain type safety end to end the approach reduces manual cleaning and lets analysts write Dataset-backed feature code that can run in structured streaming though typed ML on tensors remains unsolved

AI-written summary · about

" chapters (19)

1. 0:00Intro
2. 0:36What is care management
3. 1:43How do we do this
4. 2:37Care management
5. 3:43Data collection
6. 5:48Genomics
7. 8:16Where does data science start
8. 8:41DataFrames
9. 9:18Data Science
10. 10:01Insights
11. 10:21Diversity issue
12. 10:51Infrastructure guys
13. 12:09Kafka
14. 12:51Data scientists vs engineers
15. 13:14Nitro hiring data scientists
16. 14:08Data ingestion
17. 14:42Data scientists vs production engineers
18. 18:55hiring
19. 19:26resistance

" related talks

• 14mMaking Data Quality an integral part of developing Machine Learning and Data ProductsKetan Umare, Katrina Rogan · Data @Scale Spring 2022
• 47mKeeping the fun in Apache Spark Datasets and FPHolden Karau · Scala Days Berlin 2018
• 06mInterviewBrian M Clapper · Scale By The Bay 2018
• 35mScalable Data Science with FHIRRyan Brush · HL7 FHIR DevDays Boston 2018
• 31mOne Data Pipeline to Rule Them AllSam Kitajima-Kimbrel · PyCon AU 2017
• 04mInterviewVlad Patryshev · Scale By The Bay 2018
• 54mUntangling Healthcare with Spark and DataflowRyan Brush · Philly ETE 2016
• 04mInterviewSergei Winitzki · Scale By The Bay 2018