This course develops advanced practical skills in the design, implementation, and operation of modern data engineering workflows that support data-driven analytics and decision-making. You will learn to design and build scalable ETL (Extract, Transform, Load) and ELT pipelines for reliable data ingestion, transformation, and loading across heterogeneous data sources and platforms.
The course introduces industry-standard workflow orchestration and scheduling frameworks, enabling you to manage complex data dependencies and production-grade pipelines. Emphasis is placed on containerisation and deployment practices using Docker and Kubernetes to ensure portability, scalability, and resilience of data workflows in cloud and distributed environments.
You will also explore continuous integration and continuous delivery (CI/CD) practices tailored to data systems, applying DevOps principles within a DataOps context. Through hands-on activities and case-based learning, you will gain experience in automating testing, deployment, and monitoring of data pipelines, ensuring quality, reliability, and operational efficiency in production data platforms.
On completion, you will be equipped with the technical and operational skills required to develop and maintain robust, scalable data workflows aligned with contemporary industry practices in data engineering and DataOps.