Omega and Firmament: flexible, scalable and heterogeneity-aware cluster scheduling

Опубликовано 21 июня 2016, 18:46

Increasing scale and the need for rapid response to changing requirements are hard to meet with current monolithic cluster scheduler architectures. Omega is a novel architecture for running multiple cluster schedulers atop a shared infrastructure, addressing these needs using parallelism, shared state, and lock-free optimistic concurrency. In this talk, I will compare the Omega approach to existing cluster scheduler designs, and evaluate the impact of scheduler interference. Using real Google workloads, I will show that a shared-state approach is viable, and that it enables custom scheduling logics, such as a MapReduce scheduler opportunistically making use of spare resources. I will also describe some newer, on-going work: the Firmament scheduler, which adapts a Quincy-style min-cost max-flow optimization to make good scheduling decisions in the face of heterogeneous cluster resources and workload interference.