Interview
summary
Works at DataStax and describes concrete engineering pain: integrating TinkerPop/Cassandra distributions and subtle distribution differences that break pipelines. Calls out a specific Spark scaling issue — Spark shuffle behavior forces teams to define number of partitions before running large queries, causing failures at scale. Predicts IoT explosion where current aggregation hides detail but machine learning demands raw IoT data, creating storage and query bottlenecks that projects must solve.
AI-written summary · about
related talks
40mScaling with Apache Spark (or a lesson in unintended consequences)
21mChallenges of Spark Application coexisting with NoSQL databases
50mChallenges that everyone struggles while productionizing Apache Spark workloads
57mMulti Tenant NoSQL and NewSQL Cloud data Design Patterns Deep dive
05mBret Piatt interviewed at Velocity 2010
10mInterview
41mOptimising Apache Spark and SQL for Improved Performance
54mConnect and give sense to your big data with Apache TinkerPop and graph databases