ZS Associates: Building a Self-Service Enterprise Data Engineering Platform

Опубликовано 3 января 2023, 18:09

ZS Associates is a global professional services firm providing strategy, analytics, and technology consulting to various clients across industries. Learn how ZS Associates have built a robust self-service, secure and cost-efficient data engineering platform on AWS hosting hundreds of microservices, processing millions of API and hundreds of data access requests per month.
- The architecture leverages Amazon EKS for hosting the microservices for automation workflows, UI portals & reusable data engineering jobs.
- The platform portal is leveraged to launch a variety of infrastructures for applications like batch ETL pipelines, real-time streaming pipelines, custom analytics applications, etc.
- Their automated access management workflows use Amazon S3 for storing data, Glue as a central metadata repository, IAM to control access to data sets, and Apache Ranger for fine grain access.
- The self-service data engineering platform built on AWS helped reduce the duration of their data governance workflows from weeks to a few hours, Infrastructure provisioning to a few minutes & development effort of building new data engineering pipelines by 30%.

Check out more resources for architecting in the #AWS​​​ cloud:
amzn.to/2ZIbygO​

#AWS #AmazonWebServices #CloudComputing #ThisIsMyArchitecture