Senior Data Engineer

Key Responsibilities

• Data Pipeline Architecture and Delivery: Lead the design and implementation of maintainable batch and streaming pipelines that ingest, validate, transform and publish structured, semi-structured and unstructured data across AWS and Databricks.

• Lakehouse and Data Product Engineering: Design Bronze, Silver and Gold data layers using Delta Lake or equivalent patterns; develop reusable domain data models, curated datasets and data products with explicit ownership, definitions and service expectations.

• Source Integration and Incremental Processing: Implement reliable ingestion from databases, files, APIs and event streams using full-load, incremental, change-data-capture and replayable processing patterns appropriate to each source.

• Data Quality and Contracts: Establish schema enforcement, data contracts, automated validation, reconciliation and anomaly checks covering completeness, validity, consistency, uniqueness, timeliness and referential integrity.

• Orchestration and Deployment Automation: Build parameterised workflows using Databricks Workflows or Lakeflow and relevant AWS services; integrate automated testing, infrastructure as code and CI/CD controls across development, test and production environments.

• Metadata, Lineage and Governance: Implement technical metadata, business definitions, end-to-end lineage, classification, ownership and access controls using Unity Catalog, AWS Glue Data Catalog, Lake Formation or approved platform capabilities.

• Pipeline Monitoring and Reliability: Define dashboards, alerts, logging, operational metrics and runbooks for pipeline health, freshness, data-quality failures, latency, throughput and downstream impact; lead troubleshooting and root-cause analysis.

• Performance and Cost Optimisation: Optimise Spark and SQL workloads, file sizes, partitioning, clustering, caching, compaction, cluster configuration and workload scheduling to improve runtime, reliability and AWS/Databricks cost efficiency.

• Security and Data Protection: Apply least-privilege IAM, encryption, secrets management, environment separation, audit logging, sensitive-data controls and secure data-sharing patterns in line with applicable agency requirements.

• Technical Leadership and Quality: Set engineering standards, review designs and code, mentor engineers, manage technical debt and promote reusable components rather than project-specific pipelines.

• Stakeholder Collaboration: Work with data owners, stewards, analysts, data scientists, architects, product teams and governance stakeholders to translate analytical and operational needs into implementable data specifications.

• Documentation and Knowledge Transfer: Maintain architecture decisions, source-to-target mappings, data models, contracts, operating procedures and support documentation, and transfer knowledge to the internal team throughout the engagement.

Requirements & Qualifications

A. Educational Qualification & Experience Education: Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related technical discipline. Experience: Typically 6–10 years of relevant data or software engineering experience, including hands-on design and operation of production data pipelines and at least 2 years working with cloud-based data platforms. Candidates should demonstrate technical leadership through architecture, engineering standards, code review, mentoring or ownership of critical data services.

B. Technical & Professional Competencies Essential (Must-Have)

• Strong Python, PySpark and SQL capability, including distributed processing, complex transformations, query optimisation, debugging and automated testing.

• Hands-on experience with Databricks, Apache Spark and Delta Lake for production-scale batch or streaming data engineering.

• Hands-on AWS data engineering experience with a relevant mix of S3, Glue, Lake Formation, IAM, KMS, Lambda, Step Functions, EventBridge, CloudWatch, Redshift and MSK/Kinesis.

• Experience designing lakehouse or data-lake architectures, including layered data patterns, dimensional or domain data modelling and reusable curated datasets.

• Experience implementing incremental ingestion, change data capture, idempotency, late-arriving data handling, backfills, replay and recovery.

• Experience implementing data quality, schema validation, reconciliation, metadata and lineage as automated parts of production pipelines.

• Experience with workflow orchestration, Git-based development, CI/CD, infrastructure as code such as Terraform, and environment promotion controls.

• Ability to implement pipeline observability, operational dashboards, alerting, runbooks, incident diagnosis and service recovery.

• Working knowledge of cloud security, least-privilege access, network boundaries, encryption, secret management, data classification and audit logging.

• Ability to review technical designs and code, establish engineering standards, and guide less-experienced engineers.

Preferred (Advantageous)

• Experience with Databricks Unity Catalog, Lakeflow or Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing and Databricks Asset Bundles.

• Experience with Apache Airflow, dbt, Kafka, Amazon MSK/Kinesis, AWS DMS or equivalent ingestion and orchestration technologies.

• Experience designing data products, data contracts, semantic or context layers, master/reference data patterns or data mesh-aligned operating models.

• Experience supporting feature engineering, ML training datasets, feature stores or MLOps pipelines on Databricks and AWS.

• Experience in Singapore public-sector environments, including IM8-aligned controls, data classification, VAPT, system assurance or governed cross-agency data sharing.

• Relevant certifications such as Databricks Certified Data Engineer Professional, AWS Certified Data Engineer – Associate or AWS Certified Solutions Architect – Associate/Professional.

C. Soft Skills & Communication Attributes

• Strong analytical and systems-thinking capability, with the discipline to identify failure modes, edge cases and operational dependencies before production deployment.

• Clear written and verbal communication, including the ability to explain data architecture, quality, lineage, security, cost and delivery trade-offs to technical and management audiences.

• Collaborative working style across data engineering, platform engineering, analytics, data science, architecture, cybersecurity and governance teams.

• Ability to translate ambiguous business needs into clear data definitions, contracts, acceptance criteria and incremental delivery plans.

• Strong ownership and production mindset, including careful documentation, support readiness and follow through on incidents and technical debt.

• Ability to mentor engineers constructively, raise engineering standards and build sustainable in-house capability rather than creating vendor dependency.