We are seeking an experienced and proactive Data Engineer to lead the development of scalable, production-grade data pipelines while mentoring junior engineers and shaping engineering standards. You will work hands-on with Databricks (including Azure Databricks), Delta Lake, and Apache Spark to support data-driven decision-making across analytics, product, and data science teams. This role combines technical depth with team leadership, ideal for someone ready to take ownership of critical data systems while guiding others.
Ready to apply for roles like this?
Unlock the company name and direct application link. Subscribers get instant access to fresh jobs across Dubai, Abu Dhabi and Riyadh, many with visa support.
Unlock employer & apply directly
Accountabilities:
- Design, develop, and optimize scalable data pipelines using Databricks and Azure Databricks, including Apache Spark and Delta Lake.
- Lead and mentor junior engineers through code reviews, architecture discussions, and technical guidance.
- Collaborate with cross-functional teams to define data requirements and deliver reliable data solutions.
- Manage workflows across Azure Data Lake Storage, Azure Synapse Analytics, and Azure Data Factory (or equivalent orchestration tools).
- Contribute to platform architecture decisions involving data modeling, processing frameworks, and performance tuning.
- Implement CI/CD pipelines for data workflows using tools like Azure DevOps or GitHub Actions.
- Enforce engineering best practices around testing, documentation, and maintainability.
- Monitor, troubleshoot, and resolve issues related to data pipeline performance and reliability.
- Ensure data governance, quality, and compliance by integrating with tools like Unity Catalog or Azure Purview.
Education & Experience:
- Bachelor's degree in computer science or a related field.
- 5+ years of experience in data engineering or a closely related technical field.
- Strong proficiency in Python and SQL for building and managing data workflows.
- Hands-on experience with both Databricks and Azure Databricks, including production use of Apache Spark and Delta Lake.
- Familiarity with cloud-based data platforms (preferably Microsoft Azure).
- Experience with Azure Data Factory, Azure Data Lake, and/or Azure Synapse Analytics.
- Demonstrated ability to lead technical projects and mentor junior engineers.
- Solid grasp of CI/CD practices and version control systems (e.g., Git).
Preferred Skills:
- Exposure to data transformation tools like dbt.
- Familiarity with data governance tools such as Unity Catalog, Azure Purview, or similar.
- Experience with containerization (e.g., Docker) and infrastructure-as-code tools (e.g., Terraform, Bicep).
- Strong communication and documentation skills; able to interface with technical and non-technical stakeholders.
