About the Job:
We are hiring for projects in the Middle East, where our team consists of frontend and backend developers, data analysts, data scientists, architects, and project managers. As a Lead Data Engineer, you will manage the development, implementation, and maintenance of data infrastructure for a modern data platform.
Responsibilities:
- Platform Design: Collaborate with cross-functional teams to refine and design the architecture of our data platform.
- Platform Optimization: Continuously enhance the performance, scalability, and reliability of the data platform to meet evolving business requirements.
- Platform Maintenance: Implement monitoring, alerting, and maintenance processes to ensure the availability and health of the data platform.
- Data Governance: Establish and enforce data governance best practices to uphold data quality, security, and compliance standards.
- Platform Integration: Integrate new technologies seamlessly into the data platform ecosystem, ensuring compatibility and interoperability.
- Collaboration: Work closely with data engineers, data scientists, analysts, and business stakeholders to comprehend requirements and offer technical expertise and support.
- Documentation: Create and maintain comprehensive documentation for the data platform architecture, processes, and workflows.
- Mentorship: Provide guidance and mentorship to junior members of the data engineering team.
Technical Responsibilities:
- Design, develop, and maintain data pipelines for data ingestion and transformation using Databricks.
- Implement declarative ETL pipelines and optimize data models and pipelines for performance, scalability, and reliability.
- Ensure data governance and security best practices are followed.
Skills Required:
- Overall 5-7 years of experience in data engineering and transformation on Cloud.
- 3+ Years of strong experience in Azure Data Engineering and Databricks.
- Experience in PySpark and developing workloads for Spark distributed computing.
- Preferably a degree in Computer Science/IT, Engineering (Computer/Telecommunication), or equivalent.
- Languages: English (C2 Proficient)
Desired Experience:
- Experience in implementing platform and application monitoring using cloud-native tools.
- Familiarity with DevOps practices for continuous integration and deployment.
Seniority: Lead
Location: Abu Dhabi Emirate, United Arab Emirates
Work Conditions: On-site, Full-time
