We are seeking a Data Engineer with expertise in data architecture, processing, and integration to support AI and machine learning initiatives. The role involves building and maintaining scalable data pipelines, ensuring data quality, and enabling seamless integration of AI models. The ideal candidate will have experience working with large-scale data platforms, cloud environments, and AI-driven applications.
Key Responsibilities:
-
Data Architecture & Engineering:
- Design and develop scalable, high-performance data pipelines for AI and ML workloads.
- Implement ETL processes to ingest, transform, and store structured and unstructured data.
- Optimize data lakes, warehouses, and real-time streaming architectures.
-
Data Integration & Processing:
- Integrate diverse data sources (e.g., APIs, relational databases, cloud storage) into AI platforms.
- Ensure data consistency, reliability, and governance across multiple environments.
- Support real-time and batch data processing for AI applications.
-
AI Model Support & Optimization:
- Collaborate with Data Scientists and ML Engineers to prepare datasets for AI/ML training and inference.
- Implement feature engineering and data transformation to enhance model accuracy.
- Support retrieval-augmented generation (RAG) for AI-driven knowledge retrieval.
-
Security & Compliance:
- Ensure data privacy, encryption, and compliance with relevant security standards.
- Implement access control and governance policies for AI-powered data solutions.
-
Collaboration & Continuous Improvement:
- Work closely with cross-functional AI teams to align data engineering with business objectives.
- Stay up-to-date with emerging AI and data engineering trends to enhance system capabilities.
- Contribute to technical documentation, best practices, and process automation.
Required Qualifications & Skills:
-
Technical Skills:
- Strong experience with Python, SQL, and Apache Spark for data processing.
- Proficiency in cloud platforms (AWS, Azure, GCP) and data pipeline tools (Airflow, Kafka, Glue, etc.).
- Expertise in database management (PostgreSQL, MySQL, NoSQL, BigQuery, etc.).
- Familiarity with ML model deployment, MLOps, and DevOps pipelines.
- Experience integrating large language models (LLMs) and AI-driven applications is a plus.
-
Soft Skills:
- Excellent problem-solving and analytical abilities.
- Strong communication and teamwork skills.
- Ability to work independently in a remote environment while maintaining high productivity.
Preferred Qualifications:
- Experience working on AI/ML-focused projects.
- Familiarity with data governance, FAIR data principles, and AI ethics.
