Why do we Build Data Centre Clusters Like We Do?

Опубликовано 18 августа 2016, 4:59

This talk will we will challenge the conventional wisdoms of how we build data center clusters. We will describe the CamCube project which has been rethinking how we build data center clusters, particularly ones that support data analytics workloads (or ΓÇ£big data jobsΓÇ¥). What happens if we try and combine the best ideas from the fields of Distributed Systems, High Performance Computing and Networking? The result of this weird alliance is the CamCube, a cluster that uses commodity hardware but is part HPC cluster and part programmable router. It openly violates most of the usual rules on how to build data center clusters. To demonstrate the benefits we will discuss the performance of a MapReduce-like data analytics platform that we run on the CamCube.