Topological space creation and Clustering at BigData scale
summary
Describes methods for discovering the intrinsic geometry of data (non‑Euclidean, possibly curved or with holes) and then performing clustering within that learned topological space. Covers analytical and machine‑learning approaches, with concrete Big Data techniques and Apache Spark code examples showing how to build topological representations and cluster at TB scale over large data lakes.
AI-written summary · about
chapters (25)
- 0:00Intro
- 1:59BigData Mining & Clustering
- 3:19Geometry and data
- 6:37Data Geometry approaches
- 6:58Topological spaces
- 9:44Euclidean space
- 10:48Curved spaces
- 13:24A complex Geodesic
- 14:10Geometries and Distances
- 15:40Gaussian Curvature of a surface
- 19:41Manifold Geometry & Topology
- 20:07Riemannian geometry
- 24:03Data analysis & Clustering: Case-5
- 28:26Manifold construction: Global vs Local
- 30:02DESCAN: Clustering on local neighborhood
- 33:00Bipartite probability graph
- 35:17Distance metrics over probability spaces
- 36:05Problem statement so far
- 36:56Understanding the clustering epsilon Frequency distribution Histogram, density plot
- 38:15Method proposed: Finding optimal dustering epsilon
- 42:09Reducing shuffle operations
- 43:57Reducing feature vector size
- 44:41Complexity tradeoffs: Time vs Space
- 45:33Distance Matrix creation: Optimization - 4
- 46:35Scala performance issues
related talks
34mTopological Data Analysis
06mComplex Data Analysis
50mThe Shape of Data: Machine Learning and Topology
28mAnthony Bak, Principal Data Scientist at Ayasdi @ MLconf SEA
26mMathematical Approaches to Clustering
40mTopological Techniques for Unsupervised Learning
43mTopological data analysis: How to quantify "holes" in your data and why?
20mAllison Gilmore, Data Scientist, Ayasdi