Role - L2 Azure Cloud Engineer
Work location - Singapore
Job Summary:
- We are seeking a mid-level Azure Cloud Engineer to design, implement, and maintain cloud infrastructure solutions for our unified data platform on Microsoft Azure. The ideal candidate will have hands-on experience with Azure data services, analytics platforms, and AI/ML capabilities to support our enterprise-wide data and analytics initiatives.
Key Responsibilities:
- Design and deploy Azure data platform solutions including Azure Synapse Analytics, Azure Data Factory, Azure Data Lake Storage (ADLS Gen2), and Databricks/Apache Spark environments.
- Build and maintain data ingestion pipelines using Azure Data Factory and Azure Synapse pipelines to support batch and real-time data processing. Implement and manage Azure Databricks workspaces and Apache Spark clusters for large-scale data processing and analytics workloads.
- Configure and optimize Azure Synapse Analytics dedicated SQL pools, serverless SQL pools, and Spark pools for data warehousing and analytics. Deploy and manage Azure Machine Learning workspaces, compute instances, and ML pipelines for model training and deployment.
- Implement Azure OpenAI service integrations for generative AI and LLM-based applications within the data platform. Design and implement data lake architecture following medallion architecture (bronze, silver, gold layers) for data organization and governance.
- Monitor data platform resources using Azure Monitor, Log Analytics, and service-specific monitoring tools to ensure optimal performance and availability. Manage Azure Active Directory, role-based access control (RBAC), and data security policies including encryption, network security, and private endpoints.
- Troubleshoot and resolve L2 escalations related to data pipeline failures, Spark job performance issues, and data platform infrastructure. Implement backup, disaster recovery, and business continuity solutions for critical data assets.
- Optimize cloud costs by analyzing resource utilization across data services and implementing cost-saving measures like autoscaling and appropriate sizing.
- Collaborate with data engineers, data scientists, and analytics teams to support CI/CD pipelines using Azure DevOps.
- Build and maintain Azure DevOps pipelines for automated deployment of Terraform configurations across development, staging, and production environments.
- Implement Azure Devops workflows for infrastructure changes, including pull request reviews, automated testing, and approval gates in Azure DevOps.
- Develop and maintain Infrastructure as Code (IaC) using Terraform to provision and manage Azure data platform resources in a repeatable and version-controlled manner.
- Manage Terraform state files securely using Azure Storage with state locking and implement modular Terraform code structure for reusability.
- Maintain comprehensive documentation for data platform architecture, standard operating procedures, runbooks, and data flows.
- Administer and maintain Linux and Windows server environments and AKS cluster.
- Administer Active Directory Domain Services (AD DS) infrastructure, GPO, DNS services.
- Execute monthly patching cycles for both Linux and Windows servers.
Required Qualifications:
- 4-8 years of experience working with Microsoft Azure cloud services, with focus on data and analytics platforms.
- Strong hands-on experience with Azure Synapse Analytics, Azure Data Factory, and Azure Data Lake Storage.
- Proficiency with Apache Spark and distributed data processing concepts. Experience deploying and managing Azure Databricks environments.
- Knowledge of Azure Machine Learning services and ML lifecycle management. Familiarity with Azure OpenAI service and integration patterns.
- Understanding of data warehousing concepts, ETL/ELT processes, and data modeling. Experience with data lake architecture patterns and best practices.
- Proficiency in PowerShell, Azure CLI, Python, or PySpark for automation and data processing. Understanding of networking concepts including private endpoints, VNets, NSGs, and service endpoints for secure data platform deployments.
- Experience with Infrastructure as Code tools like ARM templates, Bicep, or Terraform. Microsoft Azure certifications such as AZ-104 (Azure Administrator), DP-203 (Azure Data Engineer), or AZ-305 (Azure Solutions Architect) preferred.
- Strong troubleshooting and performance tuning skills for data workloads. Excellent communication and documentation abilities.
- Design and implement enterprise networking solutions including hub-and-spoke topology, Azure Virtual WAN, Azure Firewall, and private endpoints, VPN Gateway network security groups (NSGs), application security groups (ASGs) etc.
Preferred Skills:
- GCC experience / knowledge is preferred.
- Familiarity with Delta Lake and lakehouse architecture patterns. Experience with MLOps practices and automated ML model deployment.
- Understanding of AI/LLM concepts and prompt engineering for Azure OpenAI. Experience with data visualization tools like Power BI integration with Azure services.
- Knowledge of security best practices for data platforms and compliance frameworks (GDPR, HIPAA, etc.).
- Experience with containerization (Docker, Kubernetes/AKS) for ML workloads, Understanding of DevOps practices and CI/CD for data pipelines.
