Data Engineers Shouldn’t Suffer, or Some Things About Big Data Tools for IntelliJ IDEA
summary
A demonstration of the Big Data Tools IntelliJ plugin that integrates Spark, Hadoop/YARN and Apache Zeppelin into the IDE so data engineers can edit, run and monitor jobs without switching to multiple browser tabs. The plugin shows live features including remote filesystem browsing, partial Parquet file previews, YARN job logs and stack traces linked to source code, and a Zeppelin variable inspector that previews runtime DataFrame contents; demos run end-to-end and the plugin is in closed testing.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 4:24Data engineers' problems
- 8:25Zeppelin vs Spark challenges
- 12:42Data processing mistakes
- 18:55Big Data Tools plugin
- 21:45Plugin demo
- 25:40Target audience
related talks
35mExploratory Data Analysis with Spark and IntelliJ IDEA
41mApache Zeppelin, the missing GUI for your Big Data backend
43mApache Zeppelin, the missing GUI for your Big Data back-end
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding
50mIntelliJ IDEA Tips and Tricks
52mIntelliJ IDEA Tips and Tricks
51mIntelliJ IDEA-Based IDE Tips & Tricks
43mBoost Your Productivity with IntelliJ, Scala Plugin and More