Responsibilities :

Define and execute testing strategies for Generative AI/LLM systems, covering functional, non-functional, and safety-focused validation.

Design evaluation frameworks and test suites using tools such as DeepEval to measure quality, robustness, and consistency of model outputs.

Plan and conduct red teaming exercises to identify vulnerabilities (e.g., prompt injection, jailbreaks, harmful content, data leakage) and recommend mitigations.

Establish Responsible AI assurance checks aligned to fairness, transparency, privacy, and safety expectations.

Create clear test plans, evidence, dashboards, and reports that communicate risks, findings, and remediation priorities to stakeholders.

Collaborate with engineering and product teams to integrate AI testing into delivery pipelines and improve release readiness criteria.

Support incident triage and root-cause analysis for model behavior issues, regressions, and evaluation drift across versions.

Additional Responsibilities:

Proven experience executing red teaming for LLM applications and translating findings into actionable controls and mitigations.

Practical experience implementing Responsible AI practices and assurance workflows across the AI lifecycle.

Strong understanding of LLM failure modes (hallucinations, toxicity, bias, prompt sensitivity) and methods to test and reduce them.

Experience building automated evaluation pipelines and regression suites for LLM systems using DeepEval or similar frameworks.

Consulting experience: stakeholder management, requirement discovery, and delivering clear outcomes under ambiguity.

Technical and Professional Requirements:

BTech/BE or equivalent technical degree.

3â€9 years of experience in testing/quality engineering, with hands-on exposure to Generative AI or LLM testing initiatives.

Working knowledge of AI testing concepts, including test design, evaluation metrics, and result interpretation for LLM outputs.

Experience contributing to structured documentation such as test plans, test evidence, and defect/risk reporting.

RPA-GenAI Trust and Assurance Consultant