How we abused PHP and Laravel to do Big Data.
summary
A homegrown system called Cube uses PHP and Laravel to do ETL, object‑based consolidation and to export bulk slices to BigQuery for true big‑data queries. Runtime issues force tiny job chunking, stateless workers and CSV + LOAD DATA imports to beat slow writes; a custom query builder runs 10–20× faster than Eloquent. Migrating from Hadoop cuts infra cost to 1/8 of the previous bill; model training gets a JSON API wrapper so Python jobs only consume fully consolidated data.
AI-written summary · about
chapters (8)
- 0:00Start
- 0:42Introduction
- 6:24Why Php
- 8:14Milestones
- 13:37No Jobs Overhead
- 17:28Tesseract
- 22:22Performance Test
- 28:33Automatic Incremental Data Updates
related talks
31mBobby Bouwmann :: Surviving Large Applications
52mPHP at the Firehose Scale
12mToday's LAMP Stack
32mPromoting a data driven culture in a world of microservices
25mLess Pain with Gain, how Laravel helps you scale your Applications
36mThe MODERN Modern Data Stack: Building an Open Distributed Data Warehouse Beyond...
45mPHP is the king ...
49mLaravel - If, how and why you should use it!