Troubleshoot streaming job not scaling down in Dataflow

Опубликовано 27 сентября 2022, 16:27

Would you like to learn how to troubleshoot autoscaling issues in Dataflow ? In particular, are you observing that your streaming jobs are not scaling down? Would you like to know more about the autotuning features in Cloud Dataflow?

Check out this first episode to learn about the Autotuning features Horizontal autoscaling, Vertical autoscaling, and Dynamic Work Rebalancing. Watch this video to learn about the possible causes for streaming jobs not scaling down and how to resolve the same. It also covers the streaming engine and its benefits.

Chapters:
0:00 - Intro
0:16 - Autotuning features
1:44 - Problem: Streaming job not scaling down
1:59 - Validation of the criteria for autoscaling
2:20 - Limitations of streaming appliance
2:34 - Example of limitations imposed by streaming appliance
3:05 - Cause for streaming job not scaling down
3:20 - Solution for the problem
3:33 - Benefits of using the streaming engine
3:57 - Documentation and conclusion

Autotuning features in Dataflow → goo.gle/3r8hCg3

Streaming Engine → goo.gle/3dM923I

Subscribe to Google Cloud Tech → goo.gle/GoogleCloudTech