Automation Test Engineer (AI & Chatbot Testing)

We are looking for an experienced Automation Test Engineer to test and improve AI-powered customer support and troubleshooting solutions. The role focuses on test automation, API testing, AI quality validation, and performance testing to ensure reliable and accurate customer experiences.

Key Responsibilities:

• AI & Chatbot Testing

  • Develop automated tests for AI chatbots and Retrieval-Augmented Generation (RAG) systems.
  • Validate response accuracy, relevance, and AI performance.
  • Measure AI quality metrics such as hallucination rates and answer relevance.
  • Perform regression testing when AI models or knowledge bases are updated.

• Automation & Functional Testing

  • Automate testing of troubleshooting workflows and decision trees.
  • Validate end-to-end customer support scenarios.
  • Simulate customer interactions and verify expected outcomes.

• API & Integration Testing

  • Build and execute automated API tests.
  • Validate integrations between AI platforms and backend systems.
  • Create mock services and test data where required.

• Performance & Security Testing

  • Conduct load and performance testing of conversational AI systems.
  • Test AI guardrails, prompt injection risks, and unsafe responses.
  • Identify performance and reliability issues.

• CI/CD & Quality Assurance

  • Integrate automated tests into Azure DevOps pipelines.
  • Track quality metrics and generate test reports.
  • Work closely with developers, AI engineers, and product teams.

Requirements:

• 4+ years of experience in QA Automation or SDET roles.
• 1+ year of experience testing AI, Chatbots, LLMs, or Agentic AI solutions.
• Strong programming skills in Python, TypeScript, or C#.
• Experience with REST APIs, GraphQL, WebSockets, Postman, and WireMock.
• Experience with Azure DevOps and CI/CD pipelines.
• Knowledge of RAG, semantic search, and vector databases.
• Experience with MQTT or IoT telemetry is a plus.
• Exposure to embedded systems or consumer electronics is preferred.

Language Requirements:
Fluency in English is required.