Data Engineer (Pyspark)

About the Job: Join as a Data Engineer specialized in PySpark, where you will design, develop, and maintain highly scalable and optimized ETL pipelines on the Cloudera Data Platform. Your expertise in big data technologies will ensure data integrity and accuracy while supporting analytical needs across various business requirements.

Responsibilities:

• Data Pipeline Development: Design and maintain ETL pipelines using PySpark on Cloudera, ensuring high data integrity.
• Data Ingestion: Manage processes for ingesting data from diverse sources into data systems.
• Data Transformation and Processing: Utilize PySpark to process and cleanse large datasets for analytical purposes.
• Performance Optimization: Optimize PySpark code and Cloudera components to enhance performance and resource utilization.
• Data Quality and Validation: Implement monitoring and validation routines to maintain data reliability.
• Automation and Orchestration: Automate workflows with tools like Apache Oozie or Airflow within the Cloudera ecosystem.

Technical Skills:

• Experience: 3+ years in data engineering with a focus on PySpark and the Cloudera Data Platform.
• PySpark Proficiency: Advanced skills in managing RDDs and DataFrames.
• Cloudera Expertise: Strong understanding of Cloudera Manager, Hive, Impala, HDFS, and HBase.
• Data Warehousing Knowledge: Familiarity with ETL best practices and SQL tools.
• Big Data Technology: Experience with Hadoop and Kafka.
• Orchestration: Proficient in Apache Oozie, Airflow, or similar tools.
• Scripting Skills: Strong scripting abilities in Linux.

Work Conditions: Hybrid, Full-time

Location: Dubai, United Arab Emirates

About Virtusa: Virtusa is one of the fastest-growing IT Services companies in the Middle East, engaging with leading firms across different sectors, including Banking, Travel, and Telecom. The company promotes teamwork, professional growth, and an exciting work environment concentrated on advanced technology.