Objectives
In this course, you will learn how to:
• Understand the foundational roles and key concepts of data engineering, including data personas, data discovery, and relevant AWS services.
• Identify and explain the various AWS tools and services crucial for data engineering, encompassing orchestration, security, monitoring, CI/CD, IaC, networking, and cost optimization.
• Design and implement a data lake solution on AWS, including storage, data ingestion, transformation, and serving data for consumption.
• Optimize and secure a data lake solution by implementing open table formats, security measures, and troubleshooting common issues.
• Design and set up a data warehouse using Amazon Redshift Serverless, understanding its architecture, data ingestion, processing, and serving capabilities.
• Apply performance optimization techniques to data warehouses in Amazon Redshift, including monitoring, data optimization, query optimization, and orchestration.
• Manage security and access control for data warehouses in Amazon Redshift, understanding authentication, data security, auditing, and compliance.
• Design effective batch data pipelines using appropriate AWS services for processing and transforming data.
• Implement comprehensive strategies for batch data pipelines, covering data processing, transformation, integration, cataloging, and serving data for consumption.
• Optimize, orchestrate, and secure batch data pipelines, demonstrating advanced skills in data processing automation and security.
• Architect streaming data pipelines, understanding various use cases, ingestion, storage, processing, and analysis using AWS services.
• Optimize and secure streaming data solutions, including compliance considerations and access control.