Role: Gen AI Testing SME
Experience - 8-15 yrs
Responsibilities
Gen AI Testing SME
Test Strategy & Planning
- Define comprehensive testing strategies tailored for Gen AI models (LLMs, diffusion models, etc.).
- Identify key testing dimensions: accuracy, relevance, coherence, bias, toxicity, hallucination, and safety.
- Develop test plans for different stages: pre-training, fine-tuning, prompt engineering, and deployment.
Test Case Design & Automation
- Design test cases for both deterministic and non-deterministic outputs.
- Create benchmark datasets and golden sets for evaluation.
- Develop automated testing pipelines using tools like LangChain, PromptLayer, or custom frameworks.
Evaluation Metrics & Analysis
- Define and apply appropriate evaluation metrics (e.g., BLEU, ROUGE, perplexity, factual consistency).
- Analyze model outputs for hallucinations, bias, and harmful content.
- Conduct A/B testing and human-in-the-loop evaluations.
Prompt & Scenario Testing
- Test prompt robustness across variations, edge cases, and adversarial inputs.
- Validate prompt templates and chaining logic in RAG or agent-based systems.
- Ensure consistency and reliability across different user intents and contexts.
Risk & Compliance Testing
Tooling & Infrastructure
Collaboration & Reporting
