Transforming to a Data Business - Dow Jones DNA (Cloud Next '19)

Опубликовано 13 апреля 2019, 0:45

It is our hope that our lessons learned will spare other engineers restless nights tossing and turning over infrastructure considerations in transitioning to a data business. Dow Jones DNA (Data, News, and Analytics) platform was the answer to evolving customer requests to leverage premium news for text mining, machine learning, and AI solutions.

This talk will detail our decisions made as we migrated our 30-year archive of big data to Google Cloud Storage and BigQuery. We will discuss the architecture trade-offs we faced in migrating 50 TB of historic data to the cloud as well as our ever-growing corpus ingesting 1.3 million articles daily. Dow Jones DNA required data migration, data processing ongoing at scale, and performance required in query responses. The DNA platform started with a team of two data engineers and grew to a team of five data engineers. Managed services were key in making the platform possible with a small team. We will detail the balance of real-world constraints such as small team size, performance versus cost, and the upper limits of quotas as our data ever expands.

Build with Google Cloud → bit.ly/2KaUXgA

Watch more:
Next '19 Architecture Sessions here → bit.ly/Next19Architecture
Next ‘19 All Sessions playlist → bit.ly/Next19AllSessions

Subscribe to the GCP Channel → bit.ly/GCloudPlatform


Speaker(s): Patricia Walsh, Dylan Roy

Session ID: ARC204

product: Cloud - General; fullname: Patricia Walsh, Dylan Roy; event: Google Cloud Next 2019;