Gen AI Quality Engineer

Below are the key skillsets required for the role:

Core Requirements

• 5 to 8 years of experience in software quality engineering, test engineering or a similar hands on role covering complex applications.

• Strong experience testing web user interfaces, backend services and REST APIs.

• Hands on ability with API tools and automation frameworks such as Postman, REST Assured, pytest, Playwright, Cypress, Selenium or equivalent.

• Working knowledge of Java, Python, JavaScript or TypeScript sufficient to build, review and troubleshoot test automation.

• Strong test analysis skills, including requirements review, risk assessment, boundary analysis, negative testing and traceability.

• Experience validating distributed systems and integrations, including asynchronous processing, queues, batch jobs and downstream dependencies.

• Ability to inspect logs, traces, network calls, payloads and database records to identify the actual failure point.

• Experience with Git, pull requests, CI/CD pipelines, test reporting and defect management tools.

• Understanding of security and privacy testing fundamentals, including access control, sensitive data handling, input validation and auditability.

• Strong stakeholder communication and the confidence to challenge weak designs, vague expected outcomes and premature release decisions.

• Ability to work in a fast moving environment where requirements and GenAI behaviour evolve.

Solution analysis and design expectations

• Experience analysing complex applications across user journeys, business processes, APIs, data flows and enterprise integrations.

• Ability to facilitate requirements discussions and convert ambiguous business needs into clear functional requirements, acceptance criteria and solution behaviours.

• Experience producing practical analysis artefacts such as process flows, sequence diagrams, context diagrams, interface specifications, decision tables and user stories.

• Ability to analyse solution alternatives and explain tradeoffs involving quality, cost, performance, security, operational support and delivery complexity.

• Understanding of application architecture concepts including synchronous and asynchronous integrations, event flows, authentication, authorisation, failure handling and system boundaries.

• Ability to distinguish problems that require GenAI from those better addressed through deterministic rules, search, workflow automation or conventional application logic.

• Confidence working with product, architecture, engineering, security, data and business stakeholders during discovery and solution design.

GenAI and Agentic Testing Expectations

• Practical understanding of LLM based applications, retrieval augmented generation, prompts, context windows, embeddings and tool using agents.

• Ability to test probabilistic systems using evaluation datasets, repeat runs, quality thresholds and evidence based acceptance criteria rather than brittle exact text matching.

• Experience validating grounded answers, citations, retrieval quality, hallucination risk, prompt injection resistance and safe handling of restricted or unsupported requests.

• Ability to test agent plans and execution paths, tool calls, memory and state, human approval checkpoints, fallback behaviour and termination conditions.

• Awareness of evaluation and observability tooling such as Langfuse, LangSmith, OpenTelemetry, Elastic, Splunk or equivalent.

Nice to Have

• Experience testing applications in banking, finance or another regulated enterprise environment.

• Experience with contract testing, service virtualisation, synthetic monitoring or performance testing tools.

• Experience with Kubernetes, OpenShift, AWS hosted services or containerised deployments.

• Accessibility testing experience and familiarity with WCAG based checks.

• Experience building lean quality dashboards that show release risk, defect escape patterns, flaky tests and cycle time.

• Exposure to red teaming or adversarial testing of GenAI applications.

Key Domain/ Technical Skills:

(Indicate up to 3 areas)

• UI and API Quality Engineering

• Agentic and GenAI Application Testing

• Risk Based Test Automation and Observability