About the Role
We are seeking a Senior QA Engineer to take ownership of quality across our AI products. In this role, you will test LLM-powered features end to end, construct automated test and evaluation suites, and ensure that every release maintains accuracy, safety, and reliability.
Ready to apply for roles like this?
Unlock the company name and direct application link. Subscribers get instant access to fresh jobs across Dubai, Abu Dhabi and Riyadh, many with visa support.
Unlock employer & apply directly
Responsibilities
- Define test strategies and plans for AI features across web, API, and backend.
- Evaluate model outputs for accuracy, consistency, hallucinations, bias, and safety.
- Build evaluation datasets, including edge cases and adversarial/red-team prompts.
- Develop and maintain automated UI, API, and regression tests integrated into CI/CD.
- Run regression testing on model, prompt, and configuration changes.
- Test performance, latency, error handling, and fallback behavior.
- Report defects clearly and provide release-readiness sign-off.
- Test non-deterministic outputs using scoring rubrics, LLM-as-judge, human evaluation, and defined pass/fail thresholds.
- Perform AI security testing, including prompt injection, jailbreaks, and PII/data leakage across chatbots, RAG, and agents.
- Collaborate with ML engineers and product managers to define quality metrics and acceptance criteria.
- Monitor production quality and model drift after release, where applicable.
Requirements
- 7+ years of experience in QA/software testing, with a strong test automation background.
- Proven experience testing AI/ML or LLM-based products (chatbots, RAG, agents, etc.).
- Hands-on scripting experience in Python and/or TypeScript/JavaScript.
- Familiarity with at least one UI automation framework (Playwright, Cypress, Selenium) and a test framework such as Pytest.
- Experience in API testing with Postman, REST Assured, or similar.
- Experience with LLM evaluation tools.
- Working knowledge of CI/CD and version control systems, including GitHub Actions, GitLab CI, or Jenkins; Git.
- Proficiency in test management and tracking tools like Jira, TestRail, or Zephyr.
- Excellent written English skills with a clear approach to bug reporting.
Nice to Have
- Experience in performance/load testing (k6, JMeter, Locust).
- Understanding of prompt engineering, embeddings, and vector databases.
- Exposure to AI safety, red-teaming, or content policy testing.
- Familiarity with cloud services (AWS, GCP, or Azure) and Docker.
