~/talk

DataEngBytes 2023 - MEL-T1-04 - Michael Hyatt

on YouTube 19m
Talkat DataEngBytes Melbourne 2023 · on Thursday, 2023-08-31· [data-engineering, web-platform]

speaker

name:Michael Hyatt
links:none on file

resources

video:https://youtu.be/dmlE60s8l9k
slides:none on file

summary

Demonstrates schema-on-read by immediately storing API payloads (two JSON endpoints and one XML) as raw strings/JSON plus a timestamp, then parsing later with transformations in Ascend.io pipelines into a structured table; uses Python and Spark to ingest and shows Snowflake JSON extraction via pass_json functions. Claims this removes network-failure loss, lets multiple schema versions be tried, and the demo produces reproducible structured datasets from the parked payloads so recovery and schema evolution succeed.

AI-written summary · about

chapters (6)

  1. 0:00Schema on read intro
  2. 1:46API failures and handling
  3. 6:53What schema on read means
  4. 9:03Deferring schema validation
  5. 12:14Demo: schema on read
  6. 18:23Summary

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.