We are seeking an experienced Senior Data Engineer / Data Architect with a strong background in data streaming, Apache Kafka, and Airflow to join our dynamic data engineering team. This role will involve managing and enhancing a large-scale infrastructure designed for extensive social media data scraping, integration into a data lake house, and coordination across multiple data-centric teams.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines to support AI model development.
- Build and maintain efficient and reliable data pipelines using Apache Kafka, streaming services, and Airflow.
- Coordinate with the data acquisition team to ensure seamless data flow while addressing and resolving any security vulnerabilities in the Airflow and Kubernetes setup.
- Implement data solutions for large volumes of structured and unstructured data, including videos, audio, images, and text.
- Collaborate with AI researchers, machine learning engineers, and software engineers to ensure data is ready for model training.
- Ensure data quality, integrity, and security throughout the data lifecycle.
- Optimize data processing workflows for performance and scalability.
- Validate and manage multilingual data, focusing on Arabic and English datasets, especially YouTube data.
- Oversee data scraping projects from social media platforms under API constraints.
Requirements:
- Bachelor's or master's degree in computer science, Data Engineering, or a related field.
- 8+ years of experience in data engineering with a focus on building data pipelines.
- Proficiency in data processing technologies such as Apache Spark, Airflow, Kafka, Hadoop, and cloud platforms (AWS, GCP, Azure).
- Strong programming skills in Python, Java, Go, or Scala.
- Experience with SQL and NoSQL databases.
- Excellent problem-solving skills and attention to detail.
- Strong communication and teamwork skills.
Language Requirements:
- Proficiency in Arabic and English is preferred, especially for managing multilingual datasets.
What We Offer:
- An international workforce with diverse, multicultural values.
- A fair compensation package and a substantial annual leave of 25 days.
- Additional holidays for partners with a newborn.
- Medical insurance depending on the agreement.
- Opportunities for growth and development through our internal learning and development program.
- Team building activities and events.
Location: Riyadh, Saudi Arabia (On-site)
Work Conditions: Full-time
