The company is a specialist AI and data consultancy with extensive experience implementing intelligent enterprise systems across multiple industries, particularly in financial services and banking. We are seeking a skilled AWS Glue Data Engineer to join our Data Factory Squad, responsible for migrating source systems into the Lakehouse ingestion zone. This role focuses on building scalable ingestion pipelines, optimizing performance, and ensuring compliance with architectural and data assurance standards.
Ready to apply for roles like this?
Unlock the company name and direct application link. Subscribers get instant access to fresh jobs across Dubai, Abu Dhabi and Riyadh, many with visa support.
Unlock employer & apply directly
Your Responsibilities
-
Data Ingestion Development
- Building and implementing AWS Glue jobs for Bronze layer ingestion using defined standards and templates.
- Implementing correct loading methods based on source requirements (CDC, full load, delta, snapshot).
- Designing and executing historical loading mechanisms to bring legacy data into the Lakehouse.
-
Performance Optimisation
- Optimising Glue job performance (DPU allocation, parallelization, partitioning) according to best practices.
- Collaborating with platform teams to ensure tooling and optimization alignment.
-
Migration & Automation
- Aggressively migrating source tables to Bronze layer, initially using manual approaches with standards/templates, later leveraging AI-enabled acceleration.
- Ensuring jobs are version-controlled and production deployment is automated via Git and Terraform.
-
Governance & Monitoring
- Implementing source system connectivity into CDP in collaboration with source system owners.
- Ensuring jobs comply with data contracts and are properly monitored.
- Preparing documentation and handover to operational support teams.
-
Collaboration
- Working closely with Data Architect for ingestion patterns and standards.
- Coordinating with Data Assurance Lead to apply quality checks across all jobs.
- Partnering with platform engineers for tooling and optimisation.
Requirements
- Experience in AWS Glue, PySpark, and ETL pipeline development.
- Substantial knowledge of Lakehouse architecture and Medallion design principles.
- Familiarity with CDC, delta loads, and historical data ingestion strategies.
- 5+ years experience in data engineering roles, with hands-on experience in AWS Glue.
Knowledge of
- AWS services: Glue, S3, Athena, Lambda.
- Git, Terraform for CI/CD automation.
- Data quality frameworks (e.g., Soda Core).
- Identifying ways to automate work/repetitive tasks.
- Working in a fast-paced environment and delivering aggressive migration targets.
- Collaborating and communicating with different stakeholder levels.
- Working with Jira and agile methodologies.
