Starfish: A MADDER and Self-tuning System for Big Data Analytics

Опубликовано 28 июля 2016, 22:48

Timely and cost-effective analytics over "big data" is now a key ingredient for success in businesses and scientific disciplines. The Hadoop platform (consisting of an extensible MapReduce execution engine, pluggable distributed storage engines, and a range of procedural to declarative interfaces) is a popular choice for big data analytics. Hadoop's performance out of the box can be poor, causing suboptimal use of resources, time, and money. Unfortunately, practitioners of big data analytics such as business analysts, computational scientists, and researchers often lack the expertise to tune the Hadoop platform for good performance. I will introduce Starfish, a self-tuning system for big data analytics. Starfish builds on Hadoop, while adapting to system workloads and user needs to provide good performance automatically; without any need for users to understand and manipulate the many tuning knobs in the Hadoop platform. The novelty in Starfish's approach comes from how it focuses simultaneously on different workload granularities – overall workload, workflows, and jobs procedural and declarative) – as well as across various decision points – provisioning, optimization, scheduling, and data layout. Starfish is available at: cs.duke.edu/starfish