Senior Data Engineer

As a Senior Data Engineer at AI71, you will be instrumental in developing scalable, robust, and efficient data pipelines to support the development of enterprise-grade generative AI products and intelligent agents. You will work closely with cross-functional teams to process and structure data that fuels our AI models, ensuring seamless integration and delivery of insights to enterprise clients.

Key Responsibilities:

• Data Infrastructure Development:
  Design, implement, and maintain scalable data pipelines tailored for training and fine-tuning generative AI models. Build and optimize data architectures for extracting, transforming, and loading (ETL/ELT) processes, specifically for high-dimensional and unstructured datasets.

• Data Integration and Management:
  Integrate enterprise data from diverse sources, including APIs, cloud-based storage, and proprietary databases, ensuring compliance with security and privacy standards. Develop and maintain large-scale data warehouses and data lakes to support model training and analytics.

• Collaboration:
  Partner with data scientists and AI researchers to understand data requirements for model development and evaluation. Work closely with product managers and engineers to ensure data solutions align with business and product goals.

• Performance and Optimization:
  Monitor and optimize data workflows to handle large-scale, real-time data streams used by AI agents. Ensure data systems are scalable, secure, and cost-effective for enterprise-grade applications.

• Data Quality and Compliance:
  Implement systems to ensure the quality, accuracy, and consistency of training data. Ensure compliance with enterprise data governance policies, privacy regulations (e.g., GDPR, CCPA), and AI ethics guidelines.

Required Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field.
• Proven experience in building and maintaining large-scale data pipelines and architectures.
• Proficiency with programming languages such as Python, Java, or Scala.
• Expertise in SQL and experience with distributed data processing tools like Apache Spark, Hadoop, or Kafka.
• Familiarity with cloud platforms (AWS, GCP, or Azure) and their data-related services (e.g., S3, BigQuery, or Azure Data Lake).
• Experience working with unstructured data such as text, images, or videos.

Preferred Qualifications:

• Knowledge of MLOps and AI lifecycle workflows is a plus.
• Experience with generative AI technologies and frameworks (e.g., Hugging Face, OpenAI API).
• Familiarity with enterprise data tools like Snowflake or Databricks.
• Understanding of natural language processing (NLP) concepts and techniques.

Why AI71?

• Proven performance of our large language models.
• Strong traction and adoption from the open-source community.
• Secured proprietary data to build specialized distinctive models.
• Locked large compute power to support our roadmap.
• Signed anchor clients to develop POCs and demonstrate our solutions.