Senior Software Engineer

The company is seeking a Senior Software Engineer to join our team in Dubai. This position involves building a scalable data ingestion and streaming platform that processes change data capture (CDC) events in real time. The engineer will work on integrating diverse source systems into an analytics lake, utilizing technologies such as Apache Kafka, Confluent, and Databricks.

Work Conditions: Hybrid, Full-time
Duration: 3 Months

Responsibilities:

• Design and develop streaming ingestion pipelines using Apache Spark and Databricks Auto Loader to handle real-time data from cloud storage and message queues.
• Implement change data capture (CDC) and deduplication logic using Debezium and other relevant tools.
• Ensure data quality, fault tolerance, and manage data processing optimizations.
• Collaborate on architecture and orchestration patterns for data integration.
• Implement monitoring and security measures for data processing.
• Engage with analytics teams to ensure data meets their requirements and provide documentation for data lineage.
• Participate in DevOps practices using CI/CD pipelines and manage infrastructure with Terraform.

Required Skills & Experience:

• 5–8 years of experience in designing and building data pipelines with Apache Spark or equivalent big data frameworks.
• Expertise in streaming and messaging systems, particularly with Apache Kafka, Confluent Cloud, or RabbitMQ.
• Strong understanding of relational databases (SQL Server, Oracle) and experience with change event capture.
• Proficient in programming languages such as Python, Scala, or Java; solid SQL knowledge.
• Familiarity with cloud platforms (Azure, AWS) for data storage and orchestration.

Preferred Skills:

• Experience with event-driven architectures and microservices.
• Knowledge of data modeling and Lakehouse architectures, especially Delta Lake.

Language Requirements:
No specific language requirements mentioned.