Leveraging Go for Efficient Infrastructure&Data Handling
summary
Astronomer uses Go to build a commander service plus Astro CLI, Houston API and workers that integrate seamless Kubernetes/Helm integration and run inside clusters via Kubernetes client libraries. Code uses goroutines to deploy multiple Airflow instances concurrently, managing 100% of Airflow releases and contributing 18 of the top 25 committers. System uses gRPC over HTTP with Proto-defined services and microservices to reduce latency and increase throughput; the result is smaller memory overhead, higher parallelism, and predictable, tested deployments.
AI-written summary · about
chapters (5)
- 0:00Introduction to astronomer
- 1:31Airflow and astronomer role
- 4:42Why use Go for infrastructure
- 7:33Go development best practices
- 9:56Data processing optimizations
related talks
24mAstronomer's Journey from Cloud Lock-in to OSS Independence
46mSimplifying Scalable Cloud Application Development with Go
56mWhy Airflow is a secret ingredient in MLOps
27mScaling using Go
47mUsing Go in DevOps
21mLightning Talks
36mGosoline - Golang in a modern cloud environment
31mSuccessful Go for microservices architecture