Senior Data Engineer AWS / Databricks / PySpark

Senior Data Engineer - AWS / Databricks / PySpark

Role Overview

We are seeking an experienced Senior Data Engineer to design, develop and maintain scalable data pipelines and reusable data products on an AWS and Databricks-based data platform. The role involves technical leadership in data engineering, lakehouse architecture, data quality, governance and performance optimisation to support analytics, reporting and AI/ML workloads.

Key Responsibilities

• Design, develop and optimise scalable batch and streaming data pipelines using AWS, Databricks, Python, PySpark and SQL.

• Build and maintain lakehouse architectures using Delta Lake, including Bronze, Silver and Gold data layers, curated datasets and reusable data products.

• Implement data ingestion from databases, APIs, files and event streams, including incremental loading, Change Data Capture (CDC), backfills and recovery.

• Establish automated data quality checks, schema validation, reconciliation, metadata management and data lineage.

• Develop and automate data workflows using Databricks Workflows, Lakeflow and AWS services, incorporating CI/CD, automated testing and infrastructure as code.

• Implement data governance, access controls and security measures using Unity Catalog, AWS Glue Data Catalog, Lake Formation and IAM.

• Monitor pipeline performance, troubleshoot production issues and optimise Spark/SQL workloads for scalability, reliability and cost efficiency.

• Lead technical design and code reviews, establish engineering standards and mentor junior engineers.

• Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions.

• Maintain technical documentation, data models, operational procedures and knowledge transfer materials.

Requirements

• Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related discipline.

• 6-10 years of data/software engineering experience, including at least 2 years working with cloud-based data platforms.

• Strong hands-on experience with Python, PySpark, SQL, Apache Spark, Databricks and Delta Lake.

• Proficiency in AWS data services such as S3, Glue, Lake Formation, Lambda, Step Functions, EventBridge, CloudWatch, Redshift, MSK/Kinesis, IAM and KMS.

• Experience designing data lake/lakehouse architectures, data models and production-grade batch or streaming pipelines.

• Strong understanding of ETL/ELT, CDC, incremental ingestion, data quality, schema validation, metadata and lineage.

• Experience with workflow orchestration, Git, CI/CD, automated testing and infrastructure as code, preferably Terraform.

• Knowledge of data pipeline monitoring, observability, troubleshooting and performance optimisation.

• Familiarity with cloud security, encryption, access management and data governance.

• Proven ability to provide technical guidance, conduct code reviews and mentor engineers.

Good to Have

• Experience with Databricks Unity Catalog, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing and Databricks Asset Bundles.

• Familiarity with Apache Airflow, dbt, Kafka, AWS MSK/Kinesis and AWS DMS.

• Knowledge of data contracts, data products, semantic layers, master/reference data and data mesh concepts.

• Experience supporting ML/AI data pipelines, feature engineering, feature stores or MLOps.

• Exposure to Singapore public-sector environments, including IM8, VAPT, data classification and cross-agency data governance.