Driving a Realtime Personalization Engine With Cloud Bigtable (Cloud Next '19)

Опубликовано 10 апреля 2019, 2:09

Segment builds customer data infrastructure. It helps thousands of companies collect, unify, and govern their first-party customer data from over 40 sources.

A year ago, Segment faced a major engineering challenge. They were in the midst of launching a new product designed to help companies personalize their own users' experiences: Personas.

Personas needed to satisfy two very different query patterns. On the one hand, Segment had to provide fast, millisecond access to billions of rows for real-time user personalization. On the other, they also had to run aggregate computations across hundreds of millions of rows. And it had to work across the billions of new data points ingested every single day.

It was clear that using a single datastore just wouldn’t work for these two very different use cases crunching this amount of data. In this talk, Calvin French-Owen, CTO at Segment, will explain how Segment created its own Lambda Architecture, leveraging Cloud Bigtable for fast, random reads and BigQuery for crunching large analytics datasets. He’ll also discuss the decision framework used to choose these datastores, from cost analysis with tools like Dynamo, to the specific use cases served by both BigQuery and Cloud Bigtable.

The company now stores tens of terabytes in multiple systems, querying Cloud Bigtable with less than10ms latency, and scanning tens of gigabytes every second to run their batch computations. Their new product (Personas) has had significant success since its launch six months ago and has ingested 260 billion data points to date.

Build with Google Cloud → bit.ly/2TRvTL6

Watch more:
Next '19 Databases Sessions here → bit.ly/Next19Databases
Next ‘19 All Sessions playlist → bit.ly/Next19AllSessions

Subscribe to the GCP Channel → bit.ly/GCloudPlatform

Speaker(s): Calvin French-Owen

Session ID: DBS302
product:Cloud Bigtable,Cloud Datastore; event: Google Cloud Next 2019; re_ty: Publish; product: Cloud - Databases - Cloud Bigtable; fullname: Calvin French-Owen;