Exploring the alchemy of Streaming and Solr cloud
summary
E-commerce batch reindex that took 4+ hours becomes near‑real‑time by replacing twice‑daily Solr importer with Kafka streaming, Redis for nonsearch fields, and Kafka Connect → Solr sync; Solr upgrades from 6.6.2 to 8.7 and master‑slave to SolrCloud; consumer autoscaling on Kubernetes and migration to MSK. Achieved reduced fields from 548 to 78, fixed race conditions in Kafka‑Solr connector, and sped state recovery via cold bootstrapping copying RocksDB in ~8s for 100GB.
AI-written summary · about