Data Engineer (Pyspark)

We are looking for a Data Engineer with expertise in PySpark to join our team in Dubai, UAE. In this hybrid, full-time role, you will design, develop, and maintain highly scalable and optimized ETL pipelines on the Cloudera Data Platform, ensuring data integrity and accuracy. Your responsibilities will include implementing and managing data ingestion processes from a variety of sources to the data lake or data warehouse, using PySpark for data transformation and processing, and conducting performance tuning of PySpark code and Cloudera components.

Key responsibilities include:

• Data Pipeline Development: Design and maintain ETL pipelines.
• Data Ingestion: Manage processes from various sources to the data lake/warehouse.
• Data Transformation and Processing: Process and cleanse large datasets.
• Performance Optimization: Tune PySpark code and Cloudera components.
• Data Quality and Validation: Implement monitoring and validation routines.
• Automation and Orchestration: Automate data workflows using tools like Apache Oozie or Airflow.

Technical Skills Required:

• 3+ years of experience as a Data Engineer, with a focus on PySpark and the Cloudera Data Platform.
• Advanced proficiency in PySpark, including RDDs and DataFrames.
• Strong experience with Cloudera Data Platform components: Hive, Impala, HDFS, and HBase.
• Knowledge of data warehousing concepts and ETL best practices.
• Familiarity with Big Data Technologies: Hadoop, Kafka, etc.
• Experience with orchestration frameworks like Apache Oozie or Airflow.
• Strong scripting skills in Linux.

Join Virtusa and add the Middle East to your global professional experience while working on leading Digital Transformation programs. Be part of a team that values teamwork, quality of life, and professional development, with opportunities to work on exciting projects and state-of-the-art technologies.