Automation QA Engineer

The QA Engineer will join the Channel Sales and Operations team to help ensure the reliability and quality of our AI/ML-powered B2B chatbot and the foundational platforms supporting it. This role goes beyond testing outputs you'll be working closely with engineers to ensure the end-to-end system, including cloud infrastructure and data pipelines, functions as intended.

Roles and Responsibilities

. Design and execute manual and automated test cases for GenAI platforms and chatbot systems.

. Build and maintain Python-based test automation frameworks for backend services and ML pipeline validation.

. Utilize RAGAS or similar tools to assess LLM outputs for factuality, relevance, and system performance.

. Conduct end-to-end testing from data ingestion to user-facing output.

. Validate system stability across GCP cloud components compute, storage, networking, and containers.

. Identify failures not only in chatbot answers, but also in underlying infrastructure and platform behavior.

. Collaborate with DevOps and ML engineers to triage bugs and optimize performance.

. Ensure test coverage spans across multiple deployment environments including Kubernetes clusters and cloud VMs.

Requirements

. 3+ years of QA Automation Engineering experience

. Experience with Playwright for automated application testing, including sign-in and SSO authentication flows

. Ability to design tests that validate output accuracy and system behavior across different user flows

. Experience with Python API testing using requests, Pytest, and integration into CI/CD and cloud environments

. Proficiency in writing and debugging Bash scripts used in CI/CD and cloud deployment workflows

. Experience with Cloud platforms (GCP preferred), including Kubernetes (kubectl experience is a plus), virtual machines, databases, cloud networking and storage components

. Understanding of modern cloud architecture and how distributed components interconnect in production environments

. Experience with LLM testing frameworks like RAGAS, and ability to interpret metrics such as factuality, relevance, and performance

. Experience with monitoring and observability tools

. Familiarity with the end-to-end architecture of GenAI solutions, including vector stores, retrievers, embedding models, and inference systems

No Recruitment Agency needed.