Senior AI Quality Engineer - Test Automation & AI Model Validation

DeepLight AI is a specialist AI and data consultancy with extensive experience implementing intelligent enterprise systems across multiple industries, with particular depth in financial services and banking. Our team combines deep expertise in data science, statistical modeling, AI/ML technologies, workflow automation, and systems integration with a practical understanding of complex business operations.

At DeepLight, we don't believe in "off-the-shelf" fixes. We deliver tailored AI solutions designed to integrate seamlessly into existing enterprise architectures, ensuring that innovation is both scalable and secure. From building robust data foundations to deploying sophisticated AI platforms, we empower our clients to lead in an increasingly automated world.

The AI Tester is a specialized, senior-level quality engineering position within the Testing work pillar. This role is responsible for driving end-to-end quality assurance, test automation strategy, and advanced validation frameworks across complex web interfaces, microservice APIs, and cutting-edge, AI-driven systems. Operating at the intersection of traditional software testing and advanced machine learning engineering, this position focuses heavily on validating distributed backend architectures, streaming data workflows, and Generative AI/LLM outputs to ensure exceptional systemic reliability, accuracy, and performance.

Your responsibilities as the AI Tester include:

• Designing, building, and maintaining scalable, robust end-to-end automation frameworks from scratch, utilizing Playwright as the primary automation engine across web interfaces
• Authoring and executing comprehensive API testing suites to validate distributed microservices, ensuring strict data integrity, state consistency, and schema compliance
• Designing validation strategies for asynchronous, event-driven data architectures, tracking messages and auditing system behaviors across Kafka-based streaming pipelines
• Establishing specialized testing methodologies to evaluate Generative AI and Large Language Model (LLM) outputs, assessing models for hallucination, bias, semantic accuracy, and safety constraints
• Managing, curating, and version baseline prompt validation datasets and ground-truth test collections to ensure consistent benchmarking of AI system performance
• Partnering closely with AI research engineers, product owners, and DevOps squads to integrate automated testing gates directly into modern CI/CD deployment pipelines
  
  

As an AI consultancy, our greatest asset is the expertise of our people.

Searching for QA & Testing roles that provide visa sponsorship? Connect with international employers through QA & Testing Jobs with Visa Sponsorship opportunities actively seeking talented professionals.

While technical mastery is the foundation of what we do, the ability to bridge the gap between complex data science and actionable business value is what defines your success with Deeplight.

We're looking for individuals who are not only world-class in their fields of specialism, but also compelling communicators and persuasive advocates for their own skills.

You will be the face of our firm, tasked with building trust, articulating the "why" behind your technical decisions, and effectively "selling" your vision to high-level stakeholders.

If you thrive on the challenge of presenting cutting-edge solutions as much as you do on building them, you will fit right in.

Requirements

We need you to have:

• Advanced technical capability in building automated test suites using Playwright, combined with deep proficiency in testing RESTful and gRPC APIs
• Practical knowledge of utilizing specialized AI quality tools and observability platforms such as Ragas, LangSmith, or TruLens to score and evaluate model responses
• A strong technical comprehension of microservices communication patterns, database transactions, and data integrity verification across distributed environments
• Advanced coding proficiency in TypeScript, JavaScript, or Python to write clean, modular, and maintainable test scripts
• Competence in interacting with event streaming platforms (Kafka or Azure Event Hubs) to produce, consume, and validate asynchronous message payloads
• A minimum of 6 years of experience in dedicated software quality engineering, test automation, or SDET roles, with a proven focus on modern automated architectures
• A documented history of validating complex enterprise workflows that rely heavily on Kafka message queues, event sourcing, or real-time data pipelines

Explore our comprehensive directory of visa sponsorship jobs from employers worldwide who are ready to sponsor talented international professionals.

• Hands-on experience executing tests and navigating application workloads containerized via Docker and orchestrated within Kubernetes clusters
• Practical experience integrating automated test definitions, smoke suites, and regression testing gates directly into enterprise delivery setups (e.g., GitHub Actions, Azure DevOps)
  
  

It would also be great if you have:

• Conceptual or practical familiarity with the unique data privacy, regulatory security compliance requirements, and risk environments of banking applications
• Experience utilizing performance testing utilities (such as k6, JMeter, or Locust) to evaluate API latency and system threshold capacities under stress
• A basic understanding of the broader Machine Learning Lifecycle, model registry operations, and automated dataset versioning practices (e.g., DVC)
  
  

Benefits

The benefits you'll enjoy as part of this role include:

• Competitive salary

Interested in opportunities specifically in United Arab Emirates? Discover our dedicated Visa Sponsorship Jobs in United Arab Emirates page featuring roles from top employers in this location.

• Comprehensive personal health insurance
• Visa Sponsorship for the successful individual
• Professional development and certification support
• Subscription reimbursement relating to your role
• Opportunity to work on cutting-edge AI projects
• Monthly Employee Incentive program
• Career advancement opportunities in a rapidly growing AI company
  
  

This position offers a unique opportunity to shape the future of AI implementation while working with a talented team of professionals at the forefront of technological innovation. The successful candidate will play a crucial role in driving our company's success in delivering transformative AI solutions to our clients.

At DeepLight AI, we recognise that diversity drives innovation. We are committed to fostering an inclusive environment where individuals with different thinking styles can thrive and contribute their unique strengths to our specialised AI and data solutions.

Our goal is to ensure our application and interview process is accessible, predictable, and fair for all candidates.

If you require any specific adjustments to the application process, or if you require any reasonable adjustments should you be successful in being processed to the interview stage, please do let us know. This information will be kept strictly confidential and will not impact hiring decisions.