Principal AI Engineer

Job Description
Role Summary

We are seeking a hands-on Principal AI Engineer to design, build and help production wise agentic AI systems for cybersecurity use cases.

This is an AI engineering role applied to cybersecurity. The role will define and build the agentic AI harness, control plane, model evaluation framework, AI-to-system interface layer, memory and knowledge architecture, guardrails, observability model and production standards needed to deploy AI agents safely across cyber functions.

Cybersecurity knowledge is useful, but not the primary requirement. The core requirement is deep experience building production-grade LLM, agentic AI, ML, automation or platform systems. Cyber domain expertise will be provided by SOC, incident response, vulnerability management, AppSec, cloud security, IAM, GRC, threat intelligence, red-team and security engineering SMEs.

The candidate should also have prior experience operating or supporting production systems, so they can design systems that are reliable, observable, auditable, recoverable and supportable. Day-to-day operations ma ysit with a separate AI platform, engineering or operations team.

Qualifications
Required Experience
- Strong hands-on experience building production-grade LLM, agentic AI, ML, automation or platform systems.

- Deep understanding of agent architecture, orchestration frameworks, tool calling, memory design, RAG, model routing and multi-agent workflows.

- Experience with frontier models, open-source models or both, including evaluation, benchmarking and model comparison.

- Strong software engineering background, including Python, APIs, backend services, cloud platforms, containers, CI/CD, authentication, logging and production observability.

- Experience integrating AI systems with enterprise APIs, identity systems, data platforms, workflow engines, ticketing systems, codere positories and operational tools.

- Prior experience operating or supporting production systems, including monitoring, alerting, incident response, rollback, release management, access control, cost management and post-incident review.

- Practical understanding of production failure modes suchas model drift, prompt regressions, broken tool calls, API failures, retrieval errors, permission issues, latency problems, data quality gaps, cost spikes andunsafe outputs.

- Practical understanding of AI safety risks, includinghallucination, prompt injection, insecure tool use, excessive agency, sensitive data leakage, memory poisoning, adversarial manipulation and unsafe autonomousbehaviour.

- Experience designing human-in-the-loop workflows for high-risk, regulated or security-sensitive environments.

- Ability to design for operational handover, including runbooks, support models, service ownership, observability, change control and measurable service health.
Preferred Experience
- Experience building AI agents for software engineering, code review, test generation, vulnerability discovery, workflow automation or enterprise operations.

- Experience with LangGraph, AutoGen, CrewAI, SemanticKernel, AgentSea, OpenAI Agents SDK, MCP, vector databases, graph databases or similar agentic AI tooling.

- Experience with RAG pipelines, knowledge graphs, structured retrieval, event schemas, data contracts and context engineering.

- Experience with secure connector patterns, permission boundaries, service accounts, API gateways, immutable audit logging and too lmediation.

- Experience with AI red teaming, model evaluation, AI governance, secure-by-design AI or regulated-sector AI deployment.
- Experience designing or operating simulation environments, cyber ranges, replay systems, benchmark suites or adversarial test harnesses.

- Exposure to cybersecurity, AppSec, cloud security ,DevSecOps, vulnerability management, SOC operations, incident response, threat intelligence, GRC or offensive security testing.