Our Analytical Data Journey – from Paper to Petabytes
summary
Target moves from paper to petabytes by building an analytical data platform with Atomic/core histories and APIs, a pipeline framework called Kelsa (Spark/Scala) and a layered parallel processing engine plus a data‑quality and history‑statement framework. Outcomes include a 90% drop in redundant data sets, 55% fewer joins and a 75% faster history correction (from ~1.5–2 weeks to ~36–40 hours). Operational changes add a data SRE team and XD4 reconciliation tooling; the platform largely succeeds while prompting a shift toward centralized platform governance.
AI-written summary · about
chapters (10)
- 0:00Introduction to journey
- 1:03Symptoms of data problems
- 3:29Solution and architecture
- 5:59Pipeline development framework
- 8:09Skill transition challenges
- 10:17Navigating journey bumps
- 13:50Engineering operations shift
- 14:54Outcomes and benefits
- 15:57Evaluating success
- 17:01Future evolution
related talks
58mTales of Data Architecture Evolution
17mDemocratising data analytics
34mTales of Data Architecture Evolution
49mTransforming Analytical Data with Event-Driven Architecture
29mThe Data Journey
26mRe-architecting the Data Platform with Spark on Kubernetes
31mOne Data Pipeline to Rule Them All
41mOn Data: Sky above, sand below, peace within