About the job
Job Overview:
We are seeking a skilled and experienced Data Engineer to join our dynamic team. The role involves developing and managing data warehouses on Azure and AWS, integrating data from various sources, and ensuring efficient data flow through well-designed ETL/ELT processes. Responsibilities include designing and optimizing data models, ensuring data quality and compliance, and utilizing Python, PySpark, and SQL for building data pipelines. The role also emphasizes automation, performance tuning, documentation, and collaboration with cross-functional teams to support data operations and resolve issues.
Key Responsibilities:
Data Warehouse Management:
- Design, develop, and maintain scalable data warehouse solutions on Azure and AWS platforms unifying data into one source of truth.
- Implement data integration processes to ingest data from various sources.
- Develop, implement, and maintain ETL / ELT processes to ensure data accuracy and integrity.
- Monitor and troubleshoot ETL pipelines to ensure efficient data flow.
- Ensure data availability for Power BI reports and other reporting requirements.
- Utilize best practices in coding standards, testing, and version control.
Data Modelling and Architecture:
- Design and implement data models and database structures to support business requirements.
- Optimize database architecture for performance, scalability, and reliability through indexing, partitioning, and query optimization.
Data Quality and Governance:
- Implement data quality checks and data governance standards.
- Ensure compliance with data security and privacy regulations.
Automation and Scripting:
- Automate repetitive data tasks to improve efficiency and reduce manual intervention.
- Develop scripts and tools to support data operations and workflows.
Documentation and Training:
- Document data processes, workflows, and system architecture.
- Provide training and support to other team members on data-related tools and processes.
Collaboration and Support:
- Work closely with BI and data analysts, and cross-functional teams to understand data needs and deliver solutions.
- Provide support and troubleshooting for data-related issues.
Qualifications:
Education:
Bachelor’s or Master’s degree in Computer Science, Information Technology, Data Science, or a related field.
Experience:
Minimum 5 years in a Data Engineer role.
Proven experience in building and maintaining data warehouses on Azure and AWS along with peripheral services like Databricks, AWS DMS, DataFlow, and Glue.
Expertise in Python, PySpark, and SQL.
Experience working with PostgreSQL and NoSQL DBs like MongoDB.
Experience in ingesting and analyzing data from CRM and Analytics tools like Hubspot, GA4, and Mixpanel.
Extensive experience with Power BI development and deployment.
Skills:
Strong understanding of cloud computing concepts and services.
Excellent problem-solving skills and attention to detail.
Ability to work independently and as part of a team.
Strong communication and collaboration skills.
Preferred Qualifications:
Certifications:
Relevant certifications in Microsoft Azure, Power BI, and AWS are a plus.
Additional Skills:
Experience with other data visualization tools and technologies.
Knowledge of big data frameworks and tools.
