Big Data Engineer

About the job

Erbis is a software development company with offices in the US, UK, Poland, and Ukraine. The majority of our customers come from the EU and the US. We undertake project implementation on our side or augment the clients’ in-house teams. Now, our team consists of 100+ IT professionals with expertise across domains. We help enterprises and SMBs create software solutions that make the world a better place :)

About The Role

We are looking for talented Big Data Engineers to join our dynamic team for an exciting new project. This is a unique opportunity to work on a cutting-edge initiative, building a team from scratch in a fast-paced environment. If you are passionate about big data technologies and ready for a new challenge, we want to hear from you! This long-term project requires engineers to join ASAP and relocate to the Middle East.

Requirements

• Proficient in Hadoop and the HDFS ecosystem.
• At least 3 years of experience in Data Engineering.
• Extensive familiarity with Cloudera Data Platform and its runtime components.
• Strong expertise with Apache Spark.
• Hands-on experience with one or more programming languages such as Python, Go, or Java.
• Solid foundational knowledge of computer software, including Linux operating systems, storage, network I/O, and related principles.
• Experience with computing, distributed, or big data systems (e.g., Nginx, Kubernetes, Docker, OpenStack, Kafka) is advantageous.
• Highly preferred to have good data structure and system design skills.

Responsibilities

• Oversee the setup, configuration, and maintenance of Hadoop clusters and the HDFS ecosystem.
• Ensure data integrity and availability across the Hadoop Distributed File System.
• Design, develop, and maintain scalable data pipelines.
• Implement data ingestion processes to capture and store large volumes of data efficiently.
• Collaborate with data scientists and analysts to support their data needs.
• Manage and optimize the Cloudera Data Platform and its runtime components.
• Monitor system performance and troubleshoot issues to ensure high availability and reliability.
• Develop and maintain Apache Spark applications for data processing.
• Write and maintain code in programming languages such as Python, Go, or Java.
• Apply solid foundational knowledge of computer software, including Linux operating systems, storage, network I/O, and related principles.
• Utilize experience with computing, distributed, or big data systems (e.g., Nginx, Kubernetes, Docker, OpenStack, Kafka).
• Apply strong data structure and system design skills to develop reliable solutions.