Design, build and validate agentic AI workflows, including multi‑step reasoning and tool‑use orchestration.
Analyse agent and model behaviour to assess robustness, safety, error propagation and end‑to‑end decision quality.
Integrate APIs, external tools, plugins and system connectors required for agent operations.
Develop and execute evaluation approaches for LLMs and agentic systems, including scenario tests, benchmarks and automated pipelines.
Assess retrieval components, vector databases, memory systems and workflow reliability.
Apply Responsible AI principles and regulatory expectations to model behaviour, documentation and control standards.
Partner with data science, engineering, architecture and risk teams to ensure safe and compliant AI deployment.
Collaborate with other departments data scientists to understand modelling intent, technical assumptions and workflow logic.
Contribute to AI/ML proof‑of‑concept (POC) initiatives to strengthen evaluation practices and support innovation.
Requirements
Strong understanding of machine learning and LLM architectures, including components used in agentic AI systems.
Minimum 3 years of relevant experience in AI/ML engineering, model development, model evaluation, or related technical roles.
Knowledge of multi‑step reasoning, planning loops, workflow orchestration and tool‑use frameworks.
Experience evaluating LLMs, agents and AI systems through scenario‑based testing, automated evaluations and diagnostic tracing.
Ability to assess robustness, stability, drift, failure recovery behaviours and unintended model actions.
Hands‑on experience with API integration, workflow automation and inter‑system tool‑use; familiarity with Power Automate, Power Apps, Sharepoint is a plus.
Understanding of AI governance frameworks, lifecycle controls, Responsible AI principles and regulatory expectations.
Strong prompting, debugging, observability and tracing skills to diagnose complex multi‑turn agent behaviours.
Ability to collaborate across technical, risk, governance and business teams effectively.
Proficiency in Python (especially pyspark, MLlib), Hadoop stack and modern AI/ML frameworks such as PyTorch, TensorFlow, Keras, scikit-learn; familiarity with ML lifecycle framework/tool such as MLflow; familiarity with RAG, vector databases, retrieval systems and orchestration libraries is a plus.
Familiarity with Copilot Studio is a huge plus.
Strong familiarity with Microsoft Azure, Azure Databricks and Microsoft Purview is a huge plus.
