We are looking for an experienced Automation Test Engineer to test and improve AI-powered customer support and troubleshooting solutions. The role focuses on test automation, API testing, AI quality validation, and performance testing to ensure reliable and accurate customer experiences.
Ready to apply for roles like this?
Unlock the company name and direct application link. Subscribers get instant access to fresh jobs across Dubai, Abu Dhabi and Riyadh, many with visa support.
Unlock employer & apply directly
Key Responsibilities:
-
AI & Chatbot Testing
- Develop automated tests for AI chatbots and Retrieval-Augmented Generation (RAG) systems.
- Validate response accuracy, relevance, and AI performance.
- Measure AI quality metrics such as hallucination rates and answer relevance.
- Perform regression testing when AI models or knowledge bases are updated.
-
Automation & Functional Testing
- Automate testing of troubleshooting workflows and decision trees.
- Validate end-to-end customer support scenarios.
- Simulate customer interactions and verify expected outcomes.
-
API & Integration Testing
- Build and execute automated API tests.
- Validate integrations between AI platforms and backend systems.
- Create mock services and test data where required.
-
Performance & Security Testing
- Conduct load and performance testing of conversational AI systems.
- Test AI guardrails, prompt injection risks, and unsafe responses.
- Identify performance and reliability issues.
-
CI/CD & Quality Assurance
- Integrate automated tests into Azure DevOps pipelines.
- Track quality metrics and generate test reports.
- Work closely with developers, AI engineers, and product teams.
Requirements:
- 4+ years of experience in QA Automation or SDET roles.
- 1+ year of experience testing AI, Chatbots, LLMs, or Agentic AI solutions.
- Strong programming skills in Python, TypeScript, or C#.
- Experience with REST APIs, GraphQL, WebSockets, Postman, and WireMock.
- Experience with Azure DevOps and CI/CD pipelines.
- Knowledge of RAG, semantic search, and vector databases.
- Experience with MQTT or IoT telemetry is a plus.
- Exposure to embedded systems or consumer electronics is preferred.
Language Requirements:
Fluency in English is required.
