Data Engineer (Pyspark)

As a Data Engineer at Virtusa, you will play a critical role in designing, developing, and maintaining highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform (CDP). Your responsibilities will include:

• Data Pipeline Development: Create and sustain ETL pipelines, ensuring data integrity and accuracy.
• Data Ingestion: Manage the ingestion processes from various sources (e.g., relational databases, APIs, file systems) into the data lake or data warehouse on CDP.
• Data Transformation and Processing: Utilize PySpark to cleanse and transform large datasets to fulfill analytical and business needs.
• Performance Optimization: Tune performance of PySpark code and Cloudera components to enhance resource utilization and minimize ETL runtime.
• Data Quality and Validation: Implement checks and monitoring routines to maintain data quality and reliability.
• Automation and Orchestration: Automate workflows using tools like Apache Oozie or Airflow within the Cloudera ecosystem.

Technical Skills Required:

• 3+ years of experience as a Data Engineer focused on PySpark and Cloudera Data Platform.
• Advanced proficiency in PySpark, including RDDs, DataFrames, and optimization techniques.
• Strong experience with CDP components such as Cloudera Manager, Hive, Impala, HDFS, and HBase.
• Knowledge of data warehousing concepts, ETL best practices, and SQL-based tools (e.g., Hive, Impala).
• Familiarity with Big Data technologies like Hadoop and Kafka.
• Experience with orchestration frameworks such as Apache Oozie and Airflow.
• Strong scripting skills in Linux.

Virtusa is a rapidly growing IT Services company in the Middle East, working with clients in various sectors including Banking, Financial Services, Travel, and Telecom. You will have the chance to be part of award-winning projects and a team that values professional and personal development. Join a global team of over 30,000 individuals dedicated to growth and innovation.

Work Conditions: Hybrid, Full-time

Location: Dubai, Dubai, United Arab Emirates.