Pyspark - Data Architect

Location: Dubai, Dubai, United Arab Emirates
Work Conditions: Hybrid, Full-time

About the Job:
As a Pyspark - Data Architect, you will be responsible for designing, developing, and maintaining scalable and optimized ETL pipelines, managing data ingestion, transforming and processing large datasets, and ensuring data quality. You will tune performance, automate workflows, and collaborate with teams to support data-driven initiatives while maintaining comprehensive documentation.

Role: Associate Architect
Key Skills: Pyspark, Big Data – Hadoop, Hive, Impala, Kafka

Qualifications:

• Education: Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or a related field.
• Experience: 3+ years as a Data Engineer, focusing on PySpark and the Cloudera Data Platform.
• Technical Skills:
  • Advanced proficiency in PySpark, including RDDs, DataFrames, and optimization techniques.
  • Strong experience with Cloudera Data Platform (CDP) components, including Cloudera Manager, Hive, Impala, HDFS, and HBase.
  • Knowledge of data warehousing concepts, ETL best practices, and experience with SQL-based tools (e.g., Hive, Impala).
  • Familiarity with Hadoop, Kafka, and distributed computing tools.
  • Experience with orchestration frameworks like Apache Oozie and Airflow.
  • Strong scripting skills in Linux.

Language Requirements: [Not Specified]