Analytics Orchestration at Scale with Kubernetes, Tensorflow, and Kubeflow (Cloud Next '19)

Опубликовано 26 апреля 2019, 18:48

The needs for orchestrating workloads for industrial analytics go above and beyond the standard ML pipelines. Several cases involve a combination of domain specific models such as physics based models, probabilistic models and AI/ML models. In many cases, each of these models could be written in a different programming language and/or with several different libraries within the same programming language. This necessitates the need for containerized and orchestrated workloads. In some cases, different parts of the pipeline could be owned by different groups that may not want to expose the IP.
In addition, model calibration and optimization use cases require more than a simple DAG orchestration. In many of these cases, cyclic and conditional workflows are needed. When dealing with Industrial IoT applications, hundreds if not thousands of models are normally built to represent a single asset. In addition, optimization and calibration use cases require executing the pipelines thousands of times for each optimization or calibration exercise. Hence scaling such complex pipelines is a key component of industrial analytics systems. We will showcase building and managing such complex pipelines using various GCP services such as GKE, TFX, PubSub and BigTable.


Build with Google Cloud → bit.ly/2WQC04q

Watch more:
Next '19 Data Analytics Sessions here → bit.ly/Next19DataAnalytics
Next ‘19 All Sessions playlist → bit.ly/Next19AllSessions

Subscribe to the GCP Channel → bit.ly/GCloudPlatform


Speaker(s): Alexandre Iankoulski, Fabio Nonato


Session ID: DA226
product: Cloud - Containers - Google Kubernetes Engine (GKE); fullname: Alexandre Iankoulski, Fabio Nonato; event: Google Cloud Next 2019;