Building on-Premises MLOps for ISS Columbus Ground Operations - Samo Turk

on YouTube ↗35m

" summary

On-prem MLOps for ISS Columbus processes sensitive time series telemetry without public cloud by deploying a fully automated stack (Ansible + GitLab CI) that provisions microk8s clusters (<10 nodes), installs Kubeflow notebooks and pipelines, and registers artifacts in GitLab. Storage uses OpenEBS hostPath for single-node and MayaStor replication for HA, MinIO as S3, networking via Istio with NodePort plus Nginx TLS termination, IAM via Dex→Keycloak, monitoring with Prometheus/Grafana and Loki logging, a custom operator enforces per-user quotas, and the platform reaches production use in about six months with a five-person team.

AI-written summary · about

" chapters (28)

1. 0:00Introduction and speaker background
2. 1:07Project overview and requirements
3. 3:16ISS Columbus data and challenges
4. 5:43MLOps platform constraints
5. 6:45Key platform components
6. 8:12Kubernetes distribution choice
7. 9:15Automation and deployment strategy
8. 10:18Infrastructure as Code with Ansible
9. 11:00Helm vs Customize for deployments
10. 12:05Code organization and GitLab CI/CD
11. 13:11Custom operators for resource management
12. 13:52Persistent storage solutions
13. 15:20OpenEBS for block storage
14. 16:44Object storage with MinIO
15. 17:47Networking and Ingress setup
16. 18:30Load balancing with MetalLB
17. 19:14TLS security best practices
18. 20:21User identity management
19. 21:27Monitoring with Prometheus and Grafana
20. 22:31Logging with Loki and Grafana
21. 23:37CubeFlow platform features
22. 24:39CubeFlow customization and limits
23. 25:41Debugging and development tips
24. 26:45Hardware and software compatibility
25. 27:28Testing with BATS and operators
26. 27:48PostgreSQL and huge pages
27. 28:30Key takeaways and lessons learned
28. 29:37Q&A session

" related talks

• 36mMLOps from the edge to on demand visualizationDavid Duncan · DevConf.CZ 2023
• 29mTeam Teardown: MLOps in the Cloud for IoTDanielle Dean, Mathew Salvaris, Mohan Mupiddi · TWIMLcon: AI Platforms 2021
• 32mBuild an MLOps platform on top of OpenStackTorsten Deutsch, Ferenc Kuszka, Dr. Beibel · OpenInfra Summit Berlin 2022
• 53mExperience with an end-to-end MLOps pipeline: Practical insights and 3D architectural perspectivesRené Brunner, Eric Joachim Liese · MLcon London 2026
• 36mDataOps and MLOps: NLP Development and Deployment in Kubernetes made easyIgor Stranjanac · Heapcon 2024
• 33mBuilding & Managing MLOps Pipelines in the CloudJoshua Arvin Lat · Global SRE Virtual Conference 2022
• 47mIntroduction to MLOps at the EdgeAlejandro Cantos · Conf42 Machine Learning 2023
• 33mFrom spreadsheet scheduling to Kubernetes: building an on-premise ML platformMarek Molisch, Petr Chmelař · DevConf.CZ 2025