Chameleon: Expanding Open-Source Ambari for HPC

on YouTube ↗29m

" summary

Chameleon extends Apache Ambari to provision, manage and monitor Hadoop on HPC clusters by adding Ambari extension points that install and configure the Lustre parallel filesystem (MDT/OSS/clients), synchronize user accounts with an Account Manager, and let operators switch execution between HDFS and Lustre. It adds GPU and InfiniBand resource monitoring, a TimeScaleDB-backed YARN monitoring pipeline that runs Linux perf scripts per PID, and a dynamic dashboard; demo video and source release show the system functioning.

AI-written summary · about

" chapters (16)

1. 0:00Chameleon Expanding Open-Source Ambari for HPC
2. 0:27Motivation (Trend Perspective) HPC and Bigdata is converging
3. 3:07Motivation Application Perspective Genome analysis by Our scientist Some stages of data pipeline begin to support bigdata platform
4. 5:00Ambari Overview Apache Ambari is a 100% open source platform for provisioning managing and monitoring Hadoop clusters
5. 5:32Extension Points for custom service development Ambari view is a plugin that provides a way to connect custom functions to the web UI Ambari stack defines a set of everything needed to define services such as HDFS and YARN
6. 11:59lustrefs Management Service Lustre Kernel Installation function(LustrekernelUpdater)
7. 15:07Account Management Service Hadoop does not support strong authentication by default • Hadoop supports Kerberes for that, but, causes performance
8. 15:52Hadoop-on-Lustre architecture Comparison between HDFS and Lustre
9. 16:27Related works for Hadoop-on-Lustre Xyrates • MapReduce Job shows theoretical performance gains on an appropriately designed Lustre based HIPC cluster with Infinband network Seagate's lustrels plugin
10. 17:282 Hadoop-on-Lustre execution environment Hadoop-on-Lustre execution environment • Works for diskless cluster backed by Lustre • Uses secure container configuration for multi-tenancy
11. 21:11Motivation Dynamic metrics management is required
12. 21:45YARN Application Monitoring Service Time-series data monitoring
13. 22:343 TimeScaleDB Open-Source time-series database optimized for fast
14. 24:223 Data management structure Alter Table
15. 25:42HPC Resources Monitoring Provides HPC monitoring information through web UI
16. 27:42Summary HPC and big data convergence makes the distinction between data analytics and computational science's ecosystem disappear. Chameleon is a Bigdata platform operation management system considering HPC environment Chameleon helps to merge Hadoop ecosystem and Lustres.

" related talks

• 14mDelivering the enterprise data cloudAnthony Huinker · AWS re:Invent 2019
• 32mBig Data With Open Source SolutionsIgor Khrol · Big Data Conference Europe 2023
• 38mOS-Climate & SIG-SRE: Benefiting from the Operate First Community CloudKarsten Wade, Marcel Hild · CloudOpen @ Open Source Summit North America 2022
• 10mjoint data platformDoug Cutting, Ryan Peterson · Strata + Hadoop World San Jose 2015
• 12mOpenStack for Computational workloadsJon Thor Kristinsson · OpenInfra Summit Berlin 2022
• 29mCeph, Xen, and CloudStack: Semper MeliorPatrick McGarry · Xen Project User Summit 2013
• 28mOpen Sourcing the Open Cluster Management Project and the Lessons We Can Learn for AIAugust Simonelli · KubeDay Australia 2024
• 23mNeSI's Flexible HPC-A programmable infrastructure for scienceMichal Nasiadka, Sean Matheny, Blair Bethwaite · OpenInfra Summit Berlin 2022