Senior GenAI Quality Engineer and Solution Analyst

Description

We are looking for a Senior GenAI Quality Engineer and Solution Analyst to help design, analyse and validate enterprise GenAI applications across user experience, application services, data and integration flows, and agentic behaviour.

This is not a traditional testing role. The successful candidate will work across discovery, solution design, delivery and release. They will translate business needs into clear application behaviours, challenge weak or incomplete designs, analyse end to end solution flows and build an evidence based view of application quality.

The role combines strong quality engineering with practical solution analysis. It requires someone who can move comfortably between business outcomes, user journeys, APIs, data flows, model behaviour, enterprise integrations and operational controls.

Technical Skills: Required

• UI and API Quality Engineering

• Agentic and GenAI Application Testing

• Risk Based Test Automation and Observability

Responsibilities

• Define and execute end to end test strategies covering UI workflows, backend APIs, integrations and agentic interfaces.

• Test conversational and agentic behaviour including multi turn context, tool selection, tool inputs and outputs, state transitions, retries, timeouts, handoffs, approvals and recovery from partial failure.

• Validate GenAI responses for task completion, grounding, relevance, consistency, citation behaviour and safe failure, while recognising that outputs can be non deterministic.

• Perform functional, integration, regression, exploratory, negative, resilience and basic performance testing across application layers.

• Design API tests for contracts, authentication, authorisation, validation, error handling, idempotency, rate limits and downstream failures.

• Test UI behaviour across browsers and realistic user journeys, including loading states, interrupted sessions, feedback capture, accessibility basics and clear error communication.

• Create and maintain test data, reusable test scenarios and traceable evidence suitable for enterprise release governance.

• Use logs, traces, request and response payloads and observability tools to isolate defects and distinguish application, model, data, integration and platform issues.

• Automate the tests that materially reduce cycle time, manual effort or production risk, and keep unstable or low value scenarios out of the automation suite.

• Communicate defects and quality risks clearly to engineers, product owners, GenAI specialists, security teams and business stakeholders.

• Provide an evidence based release recommendation, including known limitations, residual risks and areas requiring monitoring.

• Partner with product owners, business users, architects, engineers and GenAI specialists to define the problem, target user journeys and expected business outcomes.

• Analyse proposed GenAI use cases and determine where deterministic application logic, retrieval, workflow orchestration, tool using agents or human approval should be used.

• Translate business requirements into end to end solution flows, functional requirements, interface behaviours, decision rules, acceptance criteria and non functional requirements.

• Map interactions across user interfaces, APIs, models, prompts, retrieval components, enterprise data sources, agent tools and downstream systems.

• Analyse solution options and document tradeoffs relating to quality, complexity, cost, latency, security, data access, maintainability and operational risk.

• Identify unclear ownership, missing controls, integration assumptions, failure scenarios and operational gaps before development begins.

• Support the design of human approval, fallback, escalation and exception handling paths for agentic solutions.

• Define measurable success criteria covering business outcomes, user experience, functional correctness, response quality, latency, reliability and safe failure.

• Maintain traceability from business need through solution requirement, implementation, evaluation scenario and release evidence.

• Facilitate structured design reviews and communicate findings using process flows, sequence diagrams, interface specifications, decision tables and concise solution documentation.

Skills and Qualifications

Core Requirements

• 5 to 8 years of experience in software quality engineering, test engineering or a similar hands-on role covering complex applications.

• Strong experience testing web user interfaces, backend services and REST APIs.

• Hands on ability with API tools and automation frameworks such as Postman, REST Assured, pytest, Playwright, Cypress, Selenium or equivalent.

• Working knowledge of Java, Python, JavaScript or TypeScript sufficient to build, review and troubleshoot test automation.

• Strong test analysis skills, including requirements review, risk assessment, boundary analysis, negative testing and traceability.

• Experience validating distributed systems and integrations, including asynchronous processing, queues, batch jobs and downstream dependencies.

• Ability to inspect logs, traces, network calls, payloads and database records to identify the actual failure point.

• Experience with Git, pull requests, CI/CD pipelines, test reporting and defect management tools.

• Understanding of security and privacy testing fundamentals, including access control, sensitive data handling, input validation and auditability.

• Strong stakeholder communication and the confidence to challenge weak designs, vague expected outcomes and premature release decisions.

• Ability to work in a fast-moving environment where requirements and GenAI behaviour evolve.

Solution analysis and design expectations

• Experience analysing complex applications across user journeys, business processes, APIs, data flows and enterprise integrations.

• Ability to facilitate requirements discussions and convert ambiguous business needs into clear functional requirements, acceptance criteria and solution behaviours.

• Experience producing practical analysis artefacts such as process flows, sequence diagrams, context diagrams, interface specifications, decision tables and user stories.

• Ability to analyse solution alternatives and explain tradeoffs involving quality, cost, performance, security, operational support and delivery complexity.

• Understanding of application architecture concepts including synchronous and asynchronous integrations, event flows, authentication, authorisation, failure handling and system boundaries.

• Ability to distinguish problems that require GenAI from those better addressed through deterministic rules, search, workflow automation or conventional application logic.

• Confidence working with product, architecture, engineering, security, data and business stakeholders during discovery and solution design.

GenAI and Agentic Testing Expectations

• Practical understanding of LLM based applications, retrieval augmented generation, prompts, context windows, embeddings and tool using agents.

• Ability to test probabilistic systems using evaluation datasets, repeat runs, quality thresholds and evidence-based acceptance criteria rather than brittle exact text matching.

• Experience validating grounded answers, citations, retrieval quality, hallucination risk, prompt injection resistance and safe handling of restricted or unsupported requests.

• Ability to test agent plans and execution paths, tool calls, memory and state, human approval checkpoints, fallback behaviour and termination conditions.

• Awareness of evaluation and observability tooling such as Langfuse, LangSmith, OpenTelemetry, Elastic, Splunk or equivalent.

Nice to Have

• Experience testing applications in banking, finance or another regulated enterprise environment.

• Experience with contract testing, service virtualisation, synthetic monitoring or performance testing tools.

• Experience with Kubernetes, OpenShift, AWS hosted services or containerised deployments.

• Accessibility testing experience and familiarity with WCAG based checks.

• Experience building lean quality dashboards that show release risk, defect escape patterns, flaky tests and cycle time.

• Exposure to red teaming or adversarial testing of GenAI applications.