Harnessing the Power of Spark & Cassandra within your Spring App

on YouTube ↗54m

" summary

Shows how Spark (206 EC2 nodes sorting 100 TB sorted in 23 minutes) pairs with Cassandra using the DataStax connector to offset each other’s weaknesses: Spark adds joins/aggregation, Cassandra supplies persistence and server-side filtering. Describes running one Spark executor and one Cassandra process per node to exploit data locality via per-node Spark+Cassandra, tuning cores/memory, and jar/classpath packaging for remote executors. Demonstrates a Spring pipeline (RabbitMQ → Cassandra → Spark job → Postgres) that computes 30-day low/high/avg per symbol and warns of ops cost and resource requirements.

AI-written summary · about

" chapters (59)

1. 0:00Introduction
2. 0:50Relational Databases
3. 1:22Sharding
4. 1:41Who this talk is for
5. 2:43Big Data
6. 3:12Agenda
7. 3:43Spark
8. 3:57Gray Sort
9. 5:25Spark vs Hadoop
10. 6:33Spark history
11. 6:56MOSS
12. 7:25Spark Context
13. 8:28Working with Spark
14. 9:17RDD Example
15. 10:32MapReduce
16. 10:51Demo Code
17. 11:34Spark Modules
18. 11:44Spark Sequel
19. 12:18Spark Streaming
20. 13:06Spark Window Functions
21. 13:35Cassandra
22. 15:08Token Rings
23. 16:35quorum
24. 18:12Replication across data centers
25. 18:41Why use Cassandra
26. 19:27Linear scalability
27. 21:02What is Cassandra
28. 23:37Tables
29. 24:51Terminus
30. 25:53Query
31. 26:44Data Model
32. 26:54Partition Keys
33. 27:41Clustered Rows
34. 28:57Spark vs Cassandra
35. 29:12Weaknesses
36. 30:21Companies
37. 31:04Spark Cassandra Connector
38. 32:17Spark Cassandra Smart Algorithms
39. 33:18Why Spark Cassandra
40. 33:53Spark Cassandra Example
41. 35:33How many nodes should you use
42. 37:38Common data pipeline patterns
43. 38:43Spring Cloud Dataflow
44. 39:30Programming Model
45. 40:17Code
46. 41:00Spark Configuration
47. 42:00Resource Management
48. 43:23Java Functions
49. 44:33Warning
50. 45:14Pricing
51. 46:11Understanding where your code lives
52. 46:41Mapping into classes
53. 47:15Mapping into local packages
54. 48:44Spring app demo
55. 50:18Spark master URL
56. 51:17Demo web
57. 52:23Demo client app
58. 53:04Demo client project
59. 54:08Thank you

" related talks

• 33mTo Spark or Not to SparkRussell Alexander Spitzer · Scale By The Bay 2019
• 52mSolving Analytical Problems using Apache SparkRohit Bhardwaj · Developer Summit 2020
• 40mSpark Analytics on Cassandra DataTim Berglund · Jfokus 2016
• 38mReal-time analytics with Apache Cassandra & Apache SparkChristopher Batey · Berlin Buzzwords 2015
• 05mInterviewRussell Alexander Spitzer · Scale By The Bay 2019
• 59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & AkkaHelena Edelson · Philly ETE 2015
• 39mThe Sparkling Solution that Scales to IoTKen Sipe · GOTO Stockholm 2016
• 53mLord of the SparkEvgeny Borisov · JavaDay Lviv 2016