Spark SQL acceleration on FPGA using Apache Arrow and OpenCAPI

on YouTube ↗30m

" summary

A complete pipeline converts Spark SQL to Arrow-format columnar batches, passes Arrow buffers through Fletcher to an FPGA over OpenCAPI, and runs a tidy/Tidal dataflow kernel that filters by regex and reduces durations. The prototype sustains 135 million records/sec (~4 GB/s), uses ~3.5% of the FPGA for the kernel and ~20% overall, and shows an 8x overall speedup versus one CPU thread while remaining I/O bound.

AI-written summary · about

" chapters (13)

1. 0:00Introduction
2. 0:55Big data analytics big picture
3. 2:30Requirements
4. 3:55Apache Arrow
5. 4:59Fletcher
6. 6:33Tidy
7. 7:42Where are we
8. 10:17Columnar processing
9. 13:36Proof of concept
10. 18:02Threshold workflow
11. 18:50The kernel
12. 22:16Results
13. 23:43Conclusion