At Mastech Digital, we solve meaningful business problems using data, AI, and modern digital technologies. Our teams work closely with global enterprises to build solutions that create real, measurable impact. We bring deep industry expertise across Utilities & Energy, Financial Services, Healthcare, Retail, and Technology. With an AI-first mindset and a collaborative culture, we empower our people to innovate, grow, and help clients move forward with confidence.

Role: QA Engineer – AI Applications

Location: Bangalore/Chennai, India

Employment Type: Full-Time

Years of Experience: 5-6 years

Education/Qualification: Bachelor’s degree in a related field

Submit Resume To: recruitment.helpdesk@mastechdigital.com

Role Description: The QA Engineer – AI Applications must have 5-6 years of experience. For this role, you must be a QA Engineer with hands-on experience testing AI-powered and agentic applications. 

The ideal candidate understands the unique challenges of validating non-deterministic LLM outputs, RAG pipelines, and agentic workflows — and can apply modern LLM evaluation frameworks such as DeepEval, Ragas, and LangSmith to build rigorous, automated quality gates that go well beyond traditional pass/fail assertions.

Key Responsibilities:

- Design and execute test strategies for AI-powered applications including LLM-based pipelines, agentic workflows, and semantic data layers.

- Develop and maintain automated test suites using Python-based frameworks (Pytest, Selenium) for API, regression, and integration testing.

- Build prompt regression test suites using DeepEval and PromptFoo to detect model drift, hallucinations, and output quality degradation across releases.

- Define evaluation criteria for non-deterministic AI outputs using frameworks like DeepEval (G-Eval, hallucination, faithfulness, answer relevancy metrics) and Ragas (contextual precision, recall, BLEU/ROUGE variants).

- Implement LLM-as-Judge evaluation pipelines to score agent outputs automatically against ground-truth datasets.

- Validate REST API contracts, response schemas, and error handling across all AI agent and knowledge fabric endpoints.

- Test knowledge graph queries and semantic layer outputs for correctness and consistency against expected business definitions.

- Conduct performance and load testing of AI services under production-scale traffic conditions.

- Validate data quality at ingestion, transformation, and output stages across end-to-end pipelines.

- Collaborate with developers and ML engineers to embed QA practices into CI/CD pipelines.

- Document test plans, defect reports, and AI output evaluation frameworks for team and client review.

Requirements:

- 5–6 years of QA / test engineering experience with at least 2 years on AI or ML-powered applications.

- Hands-on experience with modern LLM evaluation frameworks: DeepEval, Ragas, or TruLens for automated output quality assessment and regression benchmarking.

- Experience with prompt regression and red-teaming tools such as PromptFoo or Giskard — adversarial testing, jailbreak probing, bias and toxicity scanning.

- Familiarity with LLM observability and tracing platforms such as LangSmith or Arize Phoenix for end-to-end trace inspection and evaluation tracking.

- Hands-on experience with Python-based test frameworks: Pytest, Selenium, or equivalent.

- Solid understanding of LLM behavior, prompt engineering, and how to write evaluation suites for non deterministic outputs using metric-based scoring.

- Experience testing REST APIs using Postman, REST-assured, or equivalent tooling.

- Exposure to Azure cloud services and CI/CD pipeline integration (Azure DevOps or GitHub Actions).

- Familiarity with performance and load testing tools such as Locust, JMeter, or Azure Load Testing.

- Experience with data quality validation and pipeline testing across structured and unstructured data.

- Strong analytical mindset with the ability to define and track quality benchmarks for AI system behaviour over time.

- Bachelor’s in Computer Science, Engineering, or a related discipline.

Desired Technical Skills:

- LLM Evaluation Frameworks: DeepEval (G-Eval, hallucination, faithfulness, answer relevancy, contextual precision/recall), Ragas (RAG pipeline metrics), TruLens, Braintrust

- Red-Teaming & Safety: PromptFoo (prompt regression, adversarial probing), Giskard (bias, toxicity, vulnerability scanning), jailbreak and prompt injection detection

- Observability & Tracing: LangSmith, Arize Phoenix, Weights & Biases (W&B) trace inspection, experiment tracking, evaluation dashboards

- Test Frameworks: Pytest, Selenium, unittest, BDD (Behave / Cucumber), test fixture design, hypothesis-based property testing API Testing: Postman, REST-assured, schema validation, contract testing, mock server setup

- Performance: Locust, JMeter, Azure Load Testing, latency benchmarking, throughput analysis, token-per-second profiling

- Cloud & CI/CD: Azure DevOps, GitHub Actions, Azure Container Apps test environments, Docker-based test runners, eval gates in deployment pipelines

- Data Quality: Great Expectations, data pipeline validation, schema drift detection, ground-truth dataset management for LLM eval

Mastech Digital is an Equal Opportunity Employer - All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or disability.

Telecom Communications Privacy Notice: By submitting for this position you will receive a text message from us and you agree and authorize us to send you text messages; you can reply STOP at any time to opt-out of text messaging from us.