Identifying Personal Data by Fusing Elasticsearch with Neural Networks
summary
System converts column metadata into document lookups so Elasticsearch queries retrieve candidate labels and a small neural model acts as a neural-net calibrator to turn multiple uncalibrated scores into a single probability. Solution bootstraps from ≈30,000 manually curated columns, targets a taxonomy of 504 personal data types, and scales to annotate 2 million+ columns across 100,000 datasets. Reported outcome: about 75% of internal datasets receive at least some automated annotations.
AI-written summary · about
chapters (4)
related talks
08mLeveraging data insights to increase efficiency and innovation
18mData Classification at Scale: Taming the Hydra
50mSpeeding up data labeling with neural nets: pipeline, metrics and life hacks
30mData Driven Security
21mFrom 0 to a crowd sourced neural network in a jiffy
39mData, Science: Investigating 1 Million Galaxies with Humans and TensorFlow
21mDataEngBytes 2023 - SYD-T2-04 - Suneeta Mall
34mExposing Dark Data in the enterprise with custom NLP