As a Senior Data Engineer at Virtusa, you will be responsible for designing, developing, and maintaining highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform. Your role will include implementing data ingestion processes from various sources, using PySpark for data transformation and processing, conducting performance tuning, ensuring data quality and validation, and automating data workflows. Additionally, you will collaborate with other data professionals to fulfill analytical needs and maintain thorough documentation of engineering processes.
Responsibilities:
- Design, develop, and maintain ETL pipelines using PySpark on Cloudera.
- Implement data ingestion processes from relational databases, APIs, and file systems.
- Use PySpark to process, cleanse, and transform large datasets.
- Conduct performance tuning of PySpark code and Cloudera components.
- Implement data quality checks and validation routines.
- Automate data workflows with tools like Apache Oozie or Airflow.
- Monitor pipeline performance and troubleshoot issues.
- Collaborate with data engineers, analysts, and product managers.
- Maintain documentation of data engineering processes.
Education and Experience:
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or related field.
- 3+ years of experience as a Data Engineer with a focus on PySpark and the Cloudera Data Platform.
Technical Skills:
- Advanced proficiency in PySpark, including RDDs and DataFrames.
- Strong experience with Cloudera Data Platform (CDP) components.
- Knowledge of data warehousing concepts and SQL-based tools.
- Familiarity with Hadoop, Kafka, and other distributed computing tools.
- Experience with orchestration frameworks like Apache Oozie or Airflow.
- Strong scripting skills in Linux.
Language Requirements:
- Proficiency in English is required. Additional language skills may be a plus.
About Virtusa:
Virtusa is a leading IT services company in the Middle East, focused on delivering digital transformation programs. We work with a diverse client base and members of our team are encouraged to grow professionally and personally. Virtusa is an equal opportunity employer that values inclusion and diversity.
Join us and become part of an exciting, award-winning team dedicated to high-quality solutions and development.
