AWS re:Invent 2016: Disrupting Big Data with Cost-effective Compute (CMP302)

Опубликовано 1 декабря 2016, 23:49

Amazon EC2 Spot instances provide acceleration, scale, and deep cost savings to run time-critical, hyper-scale workloads for rapid data analysis. In this session, AOL and Metamarkets will present lessons learned and best practices from scaling their big data workloads using popular platforms like Presto, Spark and Druid.

AOL will present how they process, store, and analyze big data securely and cost effectively using Presto. AOL achieved 70% savings by separating compute and storage, dynamically resizing clusters based on volume and complexity, and using AWS Lambda to orchestrate processing pipelines. Metamarkets, an industry leader in interactive analytics, will present how they leverage Amazon EBS to persist 185 TiB of (compressed) state to run Druid historical nodes on EC2 Spot instances. They will also cover how they run Spark for batch jobs to process 1-4 PiB of data across 200 B to 1 T events/day, saving more than 60% in costs.