~/talk

Large Scale Recurring Imports in Drupal 8: Architecture and Implementation Tips

Talkat DrupalCamp Twin Cities 2017 · 2017-06-22..06-25· [drupal, data-engineering]
links:none on file
slides:none on file

Drupal 8 import architecture focuses on practical recipes: use Batch and Queue APIs plus Jenkins for off‑server CI to avoid cron time limits, parse XML/JSON/CSV (QueryPath suggested), and handle encodings up front. Implement a three-queue pipeline — source links, aggregated file blobs, then per‑row safe items (example: break a CSV into 500 queue items) — so processing happens one record at a time and can run in parallel. Prefer a custom entity for performance when millions of historical fields would bloat node tables.

AI-written summary · about

  1. 0:00Start
  2. 2:44STARTING THE JOURNEY
  3. 5:51WHAT ARE THE OPTIONS?
  4. 8:18SOURCES
  5. 10:13ARCHITECTURE
  6. 13:56FINDING THE RIGHT KEY
  7. 15:11FOUNDATION OF IMPORT
  8. 19:30PULL THE TRIGGER
~/search
type to search 6,518 conferences, 143,918 speakers and 192,133 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
← →
on a talk: 5 seconds back / forward
0–9
on a talk: jump to 0%–90% of the recording
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.