As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining advanced, scalable data systems that power critical business decisions. You will lead the development of robust data pipelines, ensure data quality and governance, and collaborate across cross-functional teams to deliver high-performance data platforms in production environments. This role requires a deep understanding of modern data engineering practices, real-time processing, and cloud-native solutions.
Location: Abu Dhabi Emirate, United Arab Emirates
Work Conditions: On-site, Full-time
Key Responsibilities:
- Data Pipeline Development & Management: Design, implement, and maintain scalable and reliable data pipelines for structured, unstructured, and real-time data feeds. Ensure data integrity, timeliness, and accuracy across systems, and implement data quality tools within transformation pipelines.
- Data Processing & Optimization: Build efficient systems using techniques like data denormalization and parallel processing. Develop stream-processing applications using Apache Kafka and optimize performance for large-scale datasets.
- Cloud, Infrastructure, and Platform Engineering: Develop and deploy data workflows on AWS or GCP, using services such as S3, Redshift, and BigQuery. Utilize Docker and Kubernetes for containerized data processing tasks in production environments.
- Database Engineering: Write and optimize complex SQL queries on relational and NoSQL databases. Work with the ELK stack (Elasticsearch, Logstash, Kibana) for search and analytics.
- Data Governance & Stewardship: Implement robust data governance, access control, and stewardship policies. Establish metadata management and auditability across pipelines.
- Machine Learning & Advanced Analytics Enablement: Collaborate with data scientists to prepare and serve features for ML models, ensuring data readiness for experimentation and deployment.
- Documentation & Continuous Improvement: Maintain documentation, including technical specifications and operational procedures. Evaluate and improve the data engineering stack by adopting new technologies.
Required Skills & Qualifications:
- 8+ years of experience in data engineering within a production environment.
- Advanced knowledge of Python and Linux shell scripting for data manipulation.
- Strong expertise in SQL/NoSQL databases such as PostgreSQL and MongoDB.
- Experience with stream processing systems using Apache Kafka.
- Proficiency in Docker and Kubernetes for deploying containerized data workflows.
- Good understanding of cloud services (AWS or Azure).
- Hands-on experience with the ELK stack for scalable search and logging.
- Familiarity with AI models supporting data management.
Preferred Qualifications:
- Working knowledge of data quality tools and lineage tracking.
- Understanding of data governance frameworks and compliance protocols.
- Experience with CI/CD pipelines for data deployments.
Education & Experience:
- Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or a related field.
- Proven ability to work collaboratively with cross-functional teams including product managers and data scientists.
