Integrating Kotlin into a Poliglot JVM Environment
summary
A Kotlin machine learning library trains on Spark/Scala, reads data from S3, and ships as a self-contained JAR embedded in Clojure services; integration succeeds but surfaces concrete surprises: Kotlin ML library calls need Seq/vararg conversions and extension wrappers to call Spark APIs, UDFs require Java SAM or wrapper objects, and Kotlin collection views and primitive arrays non-serializable break Spark task shipping. JSON deserialization and shading force a Jackson workaround, and boxed number handling causes null doubles become 0 bugs that required explicit type handling.
AI-written summary · about
chapters (29)
- 0:00Intro
- 0:56Ekata - Who we are?
- 1:44Identity Check
- 2:38The machine learning lifecycle
- 3:42Our approach
- 5:37Kotlin code in the prod web-service
- 6:11Predictor (model + code)
- 7:08Kotlin code in the ML training pipeline
- 7:49Original project structure
- 9:04Evolving codebase
- 9:50Calling varargs
- 12:08Varargs summary
- 12:20Scala closures
- 14:49Seamless above Scala 2.12
- 15:25Extension helps in 2.11
- 15:43Spark also has a Java interface
- 16:24Closures summary
- 16:57Some lists are not java.io.Serializable
- 18:50Why serialization is important?
- 20:04Serializable summary
- 20:33scala.Double
- 22:194. kotlin.Double?
- 23:36Double summary
- 26:465. Clojure maps
- 27:59Maps summary
- 29:16Broken shaded extension functions
- 30:30Shading summary
- 31:02Open sourcing the ML library
- 31:45What we learned
