At Mastech Digital, we solve meaningful business problems using data, AI, and modern digital technologies. Our teams work closely with global enterprises to build solutions that create real, measurable impact. We bring deep industry expertise across Utilities & Energy, Financial Services, Healthcare, Retail, and Technology. With an AI-first mindset and a collaborative culture, we empower our people to innovate, grow, and help clients move forward with confidence.

Role: QA Engineer – AI Applications

Location: Bangalore/Chennai, India

Employment Type: Full-Time

Years of Experience: 5-6 years

Education/Qualification: Bachelor’s degree in a related field

Submit Resume To: recruitment.helpdesk@mastechdigital.com

Role Description: The QA Engineer – AI Applications must have 5-6 years of experience. We are looking for a QA Lead with deep expertise in testing AI-powered and agentic applications, who can define and drive end-to-end quality strategy, evaluation frameworks, and governance models.

The ideal candidate brings strong experience in LLM evaluation, test automation, and AI quality engineering, and can lead teams to validate non-deterministic outputs, RAG pipelines, and agentic workflows using modern evaluation approaches.

This role goes beyond traditional QA leadership — you will own AI testing strategy, define evaluation standards, establish golden datasets, and drive continuous quality improvement across the AI lifecycle.

Key Responsibilities:

- Define and lead QA strategy for AI-powered platforms, including LLMs, RAG systems, and multi-agent workflows

- Establish enterprise-grade AI evaluation frameworks using tools such as DeepEval, Ragas, LangSmith, and TruLens 

- Build and govern golden datasets and benchmark suites for consistent model evaluation and regression tracking

- Define evaluation metrics for non-deterministic outputs (hallucination, faithfulness, relevance, accuracy, latency) 

- Lead implementation of LLM-as-Judge evaluation pipelines for automated scoring and validation 

- Design and oversee prompt regression frameworks to detect drift, degradation, and model inconsistencies 

- Establish AI testing strategies for agentic workflows, multi-step reasoning, and orchestration systems

- Drive end-to-end quality validation across data, model, API, and UI layers

- Define and enforce data quality validation frameworks (ingestion → transformation → output) 

- Lead API, integration, and performance testing strategies for AI services at scale 

- Implement adversarial testing / red teaming strategies (bias, toxicity, jailbreak attempts) 

- Establish observability and tracing standards for AI systems (LangSmith, Arize Phoenix) 

- Integrate QA practices into CI/CD pipelines with automated evaluation gates 

- Define quality benchmarks, SLAs, and continuous monitoring strategies

- Lead and mentor QA engineers; build AI QA capability within the team

- Collaborate with AI engineers, data teams, and product stakeholders to drive quality-by-design principles

- Present evaluation insights, quality metrics, and risk assessments to stakeholders

Requirements:

- 8-14 years of experience in QA / test engineering, with strong exposure to AI/ML systems

- 3+ years of experience leading QA teams or defining test strategies for complex applications

- Hands-on experience with AI/LLM evaluation frameworks (DeepEval, Ragas, TruLens, LangSmith) 

- Strong understanding of LLM behavior, prompt engineering, and evaluation methodologies 

- Experience designing golden datasets, test benchmarks, and evaluation pipelines

- Expertise in API, integration, regression, and performance testing 

- Experience with Python-based testing frameworks (Pytest, Selenium, etc.) 

- Familiarity with AI observability, tracing, and monitoring tools 

- Experience with adversarial testing and AI risk validation 

- Strong analytical and problem-solving skills with an ability to define quantitative quality metrics 

- Experience working in Agile and DevOps environments

Mastech Digital is an Equal Opportunity Employer - All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or disability.

Telecom Communications Privacy Notice: By submitting for this position you will receive a text message from us and you agree and authorize us to send you text messages; you can reply STOP at any time to opt-out of text messaging from us.