Data Engineer (PySpark)

About the Job:
As a Data Engineer, you will play a pivotal role in designing, developing, and maintaining highly scalable ETL pipelines using PySpark on the Cloudera Data Platform. You will ensure data integrity and manage data ingestion processes from various sources into our data lake or warehouse. Your expertise will support analytical needs and business requirements through the transformation and processing of large datasets, while also optimizing performance and ensuring data quality.

Key Skills: PySpark, Cloudera Data Platform, Big Data - Hadoop, Hive, Kafka

Responsibilities:

• Design, develop, and maintain scalable ETL pipelines using PySpark.
• Implement and manage data ingestion processes from diverse sources.
• Process, cleanse, and transform large datasets to meet analytical needs.
• Optimize performance of PySpark code and Cloudera components.
• Implement data quality checks to ensure accuracy throughout the pipeline.
• Automate workflows using orchestration tools like Apache Oozie or Airflow.

Technical Skills:

• 3+ years as a Data Engineer, focusing on PySpark and Cloudera Data Platform.
• Proficient in PySpark, including RDDs and DataFrames.
• Experience with Cloudera Manager, Hive, Impala, HDFS, and HBase.
• Knowledge of data warehousing concepts and SQL-based tools.
• Familiarity with Big Data technologies such as Hadoop and Kafka.
• Experience with orchestration frameworks like Apache Oozie and Airflow.
• Strong scripting skills in Linux.

Location: Dubai, Dubai, United Arab Emirates

Work Conditions: Hybrid, Full-time

About Virtusa:
Virtusa is one of the fastest growing IT services companies in the Middle East, with a client base in the UAE, KSA, Qatar, and Oman. The company collaborates with leading firms across various sectors and has received recognition for its work from industry leaders like Gartner and IDC. Virtusa values teamwork, quality of life, and professional development, offering growth opportunities and exciting projects for a global workforce of over 30,000.