Lightning talk - Broadway + Sqs + Apache Tika
summary
Broadway consumes Amazon SQS messages with a custom adapter, batches them, and calls an Apache Tika HTTP extractor that returns metadata and plain text; parsed payloads upload to S3 and are recorded in a database so the next pipeline step is configurable. A production pipeline indexes Norwegian court documents, runs downstream Elasticsearch searches and user notifications, and is reported as stable after deployment. Broadway with custom SQS adapter Apache Tika HTTP extractor stores parsed payloads to S3 with temp links
AI-written summary · about
related talks
45mKEYNOTE: Announcing Broadway
59mPolyglot Pipelines with Apache Nifi
31mOne Data Pipeline to Rule Them All
24mOrchestrating Scalable Data Pipelines with Apache Toree, YuniKorn, Spark, and Airflow
47mBuilding Modern Data Streaming Apps with Open Source
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
35mStreaming and Real-Time Analytics
42mIngestion at Scale: Open Source Pipelines for Getting Data Into Open...