Lead Data Engineer - Quality & Automation

Job Purpose

At Emirates, we believe in connecting the world, to and through, our global hub in Dubai; and in constantly innovating to ensure our customers ‘Fly Better’. Join the Emirates Group IT Team in Dubai. Emirates Group IT thrives on the dynamic nature of technology. Being pioneers in aviation innovation, we're always at the forefront, pushing boundaries. We're on the lookout for exceptional IT professionals to fortify our position as leaders in the industry. Embark on a journey with the world’s largest international airline and become a vital part of our cutting-edge information and technology team as Lead Data Engineer - Quality & Automation. The Data Quality & Automation Engineering Lead is a fully-participating member of a cross-functional team working autonomously on technology development and problem resolution in the Enterprise Data & Analytics space. The role involves leading and championing quality practices along with designing and implementing data quality and automation platforms. They will also provide support and maintenance to technical analytics solutions and products that support Emirates Airlines and the Emirates Group businesses.

Job Outline

• Work with product owners, analysts, software engineers and architects to drive understanding of technical landscape and context of deliveries & refine complex functional and non-functional requirements and translate to acceptance tests.
• Support in building the technical design by contributing in the analysis of data application and ensuring quality related features are baked into the solution.
• Work on and guide engineers on problems of diverse scope where analysis of data requires evaluation of identifiable factors and select methods to validate and automate the solution.
• Drive & mentor creation of test strategies and test plans validating the core business problem and translating them into tests around code functionality, data quality, performance, and security.
• Build components & liase with teams across the organization to manage cross dependencies for generating / mocking test data for exploratory analysis and running tests.
• Build platforms for automated tests / jobs of moderate to high scope and complexity across all stages of the data pipeline.
• Conduct code reviews to check for good coding & design practices and adherence to published coding standards and guidelines.
• Build and enhance data quality rules and platforms for observability across the data pipeline.
• Perform and support team debugging complex issues, resolve blockers.
• Advocate and perform data analysis activities such as source system analysis, data modelling, data dictionary collection, data profiling and source-to-target mapping ensuring delivery on business needs.
• Advocate and participate in updating data inventories and registries as required to keep metadata and data lineage up-to-date, following agreed Data Governance standards, guidelines and principles.

Qualifications & Experience

• Qualifications & Experience: Information Technology & Other: 5+ Years, Degree or Honours (12+3 or equivalent) Degree in a relevant field such as Computer Science, Computational Mathematics, Computer Engineering or Software Engineering. Specialization or electives in a Data & Analytics field (e.g. Data Warehousing, Data Science, Business Intelligence) are nice to have.
• Experience: 5+ years of Data Engineering experience with a focus on quality & automation.
• Skills:
  • Minimum 4+ years of testing, automation, and support experience in analytics applications such as Data Lake and Data Warehouse (preferably using the Big Data stack and Microsoft Azure cloud infrastructure.
  • Worked in a team leadership role building culture and practices around quality and developing trustworthy data assets.
  • Seasoned in building data quality platforms for highly complex data architectures performing automated checks running in a CI and in production for data observability.
  • Experienced in leading teams to build enablers/practices to identify quality issues across complex data pipelines running on big data technologies and defining rules for validating the health of the data.
  • Proficient in at least 1 language (Scala / Python) for building ETL jobs in a distributed computing environment.
  • Strong SQL querying skills is required.
  • Experienced in a wide variety of testing methods and tools covering functional, performance, and security tests across individual jobs, pipelines, and end-to-end across enterprises.
  • Good understanding of data architecture concepts such as data modeling, Big Data storage, and dimensional modeling.
  • Good experience of working with jobs in data pipelines and defining metrics/measures to ensure the correctness of the data.
  • Experience with batch and real-time data ingestion/integration tools and technologies handling massive quantities of data (structured and unstructured).
  • Exposure to Spark is highly desirable - Airline industry experience is nice to have.

Technology Domain

• Key Technologies/Tools: Big Data & distributed processing: Spark, Hadoop (HDFS, Hive, H-Base, Oozie), Airflow, , Apache Nifi, Azure (ADLS, DataBricks, Azure Data Factory) Elasticsearch, AVRO / PARQUET file formats Data Analysis, Modelling and Reporting: Snowflake, SQL, Data Vault 2.0, MicroStrategy, Power BI Cloud Technologies: Microsoft Azure and Cloudera technology stacks Integration and Messaging: Streaming (e.g. Spark Streaming), SnapLogic, TIBCO, Kafka, CI/CD : GIT, Bitbucket, Jenkins, Azure DevOps, Kubernetes, docker, SonarQube, Gatling Languages: Scala, Python.

Salary & Benefits

Join us in Dubai and enjoy an attractive tax-free salary and travel benefits that are exclusive to our industry, including discounts on flights and hotels stays around the world. You can find out more information about our employee benefits in the Working Here section of our website www.emirates.com/careers. Further information on what’s it like to live and work in our cosmopolitan home city, can be found in the Dubai Lifestyle section.

Closing date: 24 May 2024