Data Engineer (Pyspark)

As a Data Engineer at Virtusa, you will focus on designing, developing, and maintaining highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform. Your role will involve ensuring data integrity and accuracy, implementing data ingestion processes from various sources, and transforming large datasets into meaningful formats for analytics and business needs. You will also be responsible for performance tuning of PySpark code, conducting data quality checks, automation of data workflows, and utilizing orchestration tools within the Cloudera ecosystem.

Key Responsibilities:

• Design, develop, and maintain ETL pipelines using PySpark.
• Implement and manage data ingestion processes from various sources (e.g., relational databases, APIs).
• Process, cleanse, and transform large datasets.
• Conduct performance tuning of PySpark code and Cloudera components.
• Implement data quality checks and monitoring routines.
• Automate data workflows using tools like Apache Oozie or Airflow.

Technical Skills:

• 3+ years of experience as a Data Engineer, with a focus on PySpark and the Cloudera Data Platform.
• Advanced proficiency in PySpark, including working with RDDs and DataFrames.
• Strong experience with Cloudera Data Platform components such as Hive, Impala, and HDFS.
• Knowledge of data warehousing concepts and SQL-based tools.
• Familiarity with Hadoop and Kafka.
• Experience with orchestration tools like Apache Oozie or Airflow.
• Strong scripting skills in Linux.

Location: Dubai, Dubai, United Arab Emirates
Work Conditions: Hybrid, Full-time

About Virtusa: Virtusa is one of the fastest-growing IT Services companies in the Middle East, working with prominent clients across the banking, financial services, travel, telecom, and enterprise sectors. We pride ourselves on teamwork, quality of life, and professional development and have received industry awards for our work. By joining Virtusa, you become part of a global team that cares about your personal growth and offers exciting projects and opportunities.