Data Engineering, Consultant

FIND YOUR BETTER AT AIA

We don't simply believe in being The Best. We believe in better - because there's no limit to how far better can take us.

We believe in empowering every one of our people to find their better - in the work they do, the career they build, the life they live and the difference they make. So that together we can support even more people - including our own - to live Healthier, Longer, Better Lives.

Position Objective:

This role is hands-on and will focus on designing, implementing, and integrating advanced data engineering with AI solutions integration across the enterprise.

Roles and Responsibilities:

Technical Leadership

• Act as the technical lead for data engineering initiatives.
• Design, build, and optimize scalable data pipelines and architectures using enterprise tools and software.
• Drive integration of API services and components together with enterprise data platforms for digital applications.

Project Delivery & Integration

• Collaborate with internal application teams to align on project deliverables, timelines, integration requirements, and implementation strategies.
• Ensure adherence to company processes for services and infra provisioning, including user access management and compliance with governance standards.
• Oversee end-to-end project execution, from design through deployment.

Operation Excellence

• Define and establish the Business-As-Usual (BAU) support model post-project implementation.
• Participate in BAU support activities as needed, ensuring smooth operations and issue resolution.
• Drive continuous improvement in data engineering practices and operational processes.

Vendor & Stakeholder Management

• Serve as the primary technical liaison with external vendors, ensuring deliverables meet quality, security, and performance standards.
• Manage vendor relationships to align with project goals, budgets, and timelines.
• Communicate effectively with stakeholders across business and technology teams.

Innovation & Best Practices

• Stay current with emerging Data Engineering technologies and assess their applicability to enterprise data solutions.
• Champion best practices in data engineering, cloud architecture, and AI integration.
• Contribute to the development of enterprise standards, frameworks, and reusable components.

Governance & Compliance

• Adhere to company governance processes for Azure service provisioning, access management, and security compliance.
• Work closely with the Group CCoE and technical counterparts for component provisioning, user access, and platform governance.

Minimum Job Requirements:

• Min 8-10 years of data engineering design and development Plan and manage end-to-end delivery and work assignments within the assigned projects.
• Excellent skills in Business user engagement and stakeholder management
• Experienced in engaging business stakeholders for requirement analysis and discussions
• Strong experience in Azure Databricks Data Engineering in the following context:

Must-Have Skills

PySpark

• Building data pipelines
• Data transformations
• Spark performance tuning

SQL

• Complex queries
• Joins, window functions
• Data modeling

Delta Lake

• MERGE (upsert)
• Time Travel
• Partitioning
• OPTIMIZE and VACUUM

Databricks Platform

• Notebooks
• Workflows/Jobs
• Cluster management
• Git integration

Lakehouse Architecture

• Bronze, Silver, Gold layers
• ETL/ELT design

Important Skills

Cloud Knowledge

• Azure Databricks (preferred for many enterprises)
• ADLS Gen2
• ADF
• Azure DevOps

Streaming

• Auto Loader
• Structured Streaming
• Kafka or Event Hub

Governance & Security

• Unity Catalog
• Access control
• Data lineage

Nice-to-Have Skills

DevOps/DataOps

• CI/CD pipelines
• Terraform
• Databricks Asset Bundles

Programming

• Python (mandatory)
• Scala (optional)

• Sound experience in execution of IT projects with high level of expertise in system application and technology implementation delivery.
• Strong communication skills with ability to communicate with stakeholders.
• Strong analytical and problem-solving skills.