Recent trends in data and machine learning technologies
summary
Survey of concrete tech shifts: open-source stacks remain centered on Apache Spark, Kafka and Flink while Apache Pulsar adoption grows for multi-source streaming; cloud vendors push Trill-style pipelines. Data lake problems get fixed by projects like Iceberg and Delta Lake supports data versioning and incremental quality improvements that handle updates and inserts. Data quality sees ML applied—HollowClean probabilistic repairs is treated as state of the art. Time-series (Timescale, InfluxDB), graph (TigerGraph), MLflow/Comet for models and rising emphasis on model governance and provenance complete the stack.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:19Strata Data Conference
- 0:51Data Ingestion & Processing
- 3:19Data Governance
- 3:53Data Quality
- 4:21Data Management and Storage
- 5:26Data Lake: original promise
- 5:48Data Lake revisited
- 7:49Machine Learning
- 9:44Managing Risk
- 10:17Strata Data NYC 2020
related talks
42mTales from the Field: The Future is Now
54mThe Future of Data Science/ML at Scale: A Look at MLflow, Delta Lake + Lakehouse
57mBACK TO THE FUTURE: EMERGING TRENDS IN DATA ENGINEERING
19mSustaining ML in the enterprise
02mThe Future of Data Science/ML at Scale
41mInfrastructure for ML Applications
50mMasterclass: Building a data stack in 2021
47mExpert Panel: Current & Future State of the Art in Real-Time Data Infrastructure