Senior Data Engineer

As a Senior Data Engineer at Avrioc Technologies, you will collaborate with multiple stakeholders to understand the needs of data practitioners and deliver scalable solutions. Your responsibilities will include defining, building, and maintaining the Data Platform and working on emerging technologies to build distributed applications. You will oversee the end-to-end development effort to ensure timely delivery of high-quality solutions that meet requirements, adhere to architectural vision, and comply with all applicable standards. Key tasks include:

• Presenting technical solutions, capabilities, considerations, and features in business terms.
• Contributing to initiatives such as Data Discovery, Data Lineage, and Data Quality.
• Building data systems, pipelines, analytical tools, and programs.
• Conducting complex data analysis and reporting on results.

Skills and Attributes:

• Strong technical expertise in data modeling, data ingestion, distributed processing, data warehousing, and ETL processes.
• Experience with Apache Kafka and Spark Streaming.
• Familiarity with various data sources, including NoSQL databases (e.g., MongoDB), MySQL, Elasticsearch, and Kafka topics.
• Proficiency in SQL, Python, PySpark, and Spark SQL, along with a solid understanding of the Databricks platform.
• Ability to develop large-scale batch and real-time data pipelines.
• Knowledge of best practices for optimizing Databricks cluster configurations (size, type, memory) and delivering optimized code.
• Experience in maintaining, monitoring, and enhancing data pipelines.
• Familiarity with visualization tools such as Kibana and QlikView.
• Exposure to Docker, Kubernetes, and GitLab.
• Experience with JIRA processes and Confluence for documentation and project management.

Qualifications:

• 8+ years of professional experience.
• Degree in Computer Science, Data Science, Mathematics, IT, or a related field.
• 5 years of experience in a data engineer or similar role.

Location: Abu Dhabi Emirate, United Arab Emirates
Work Conditions: On-site, Full-time