Large Scale Recurring Imports in Drupal 8: Architecture and Implementation Tips
summary
Drupal 8 import architecture focuses on practical recipes: use Batch and Queue APIs plus Jenkins for off‑server CI to avoid cron time limits, parse XML/JSON/CSV (QueryPath suggested), and handle encodings up front. Implement a three-queue pipeline — source links, aggregated file blobs, then per‑row safe items (example: break a CSV into 500 queue items) — so processing happens one record at a time and can run in parallel. Prefer a custom entity for performance when millions of historical fields would bloat node tables.
AI-written summary · about
