About the Job:
We are seeking a highly experienced Senior Data Engineer to join our client and play a critical role in shaping our data infrastructure and driving innovative solutions. This individual will work closely with a talented Data Science team and contribute to cutting-edge AI projects.
Location: Abu Dhabi Emirate, United Arab Emirates
Work Conditions: On-site, Full-time
Responsibilities:
-
Data Architecture and Pipelines: Design, implement, and optimize robust data pipelines to handle large-scale data ingestion, transformation, and storage. Work with real-time and batch data processing frameworks such as Apache Spark, Apache Kafka, and Apache NiFi.
-
Collaboration with Data Science and AI Teams: Partner with data scientists and AI engineers to build efficient data workflows and deliver high-quality data for model development and analytics. Develop infrastructure to support machine learning and AI model deployment.
-
Data Integration and Storage: Manage and optimize Data Lakes and data warehouses, ensuring seamless integration and accessibility. Leverage Apache Iceberg for scalable, high-performance table formats in large-scale analytics.
-
IBM Technology Stack: Work extensively with IBM products such as IBM DataStage, IBM Db2, IBM Watson, and other relevant tools. Integrate and optimize IBM technologies to enhance data management and AI-driven solutions.
-
Graph and NoSQL Databases: Design and maintain data models using Neo4j for graph database applications. Manage and query NoSQL databases like MongoDB to support diverse data structures.
-
Data Governance and Security: Ensure compliance with data security standards and best practices. Establish frameworks for data quality, metadata management, and data governance.
-
Performance Optimization: Troubleshoot and resolve data-related issues to maintain high system performance and reliability. Continuously improve system architecture to enhance scalability and efficiency.
Required Skills:
- AI Integration: Demonstrated ability to collaborate with Data Science teams on AI and ML projects.
- Programming: Strong skills in Python, Java, or Scala for data processing and pipeline development.
- Cloud Expertise: Familiarity with cloud platforms (AWS, Azure, GCP) is a plus.
Required Experience:
- Minimum of 10 years in data engineering or a related role, with proven experience in data-intensive environments and AI projects.
- Extensive experience with IBM products (e.g., IBM DataStage, IBM Db2, IBM Watson).
- Strong knowledge of Data Lakes and Apache Iceberg.
- Proficiency with Apache Spark, Apache Kafka, and Apache NiFi for large-scale data processing.
- Expertise in Neo4j and other graph databases.
- Hands-on experience with MongoDB and NoSQL databases.
Required Education:
Bachelor's or Master’s degree in Computer Science, Data Engineering, or a related field.
Language Requirements: Not specified.
