Data Test Engineer

SHELL INFOTECH PTE. LTD.

Singapore, Paya Lebar

5-8 Years

SGD 7,000 - 9,000 per month

Save

Posted 9 hours ago
Be among the first 10 applicants

Early Applicant

Job Description

Experience Requirements:
Must have QA/Testing Experience.
Data Testing Experience specifically in Big Data, Hadoop, or Cloud Data Warehouse environments.
Databricks Experience of testing pipelines within a Databricks environment.
Automation Focus: Proven track record of moving from manual SQL checks to automated Python-based testing frameworks.
Mandatory: Databricks Certified Data Engineer Associate (at minimum).
Preferred: ISTQB Foundation or Advanced Level (Test Automation Engineer).
Core Technical Skills:
Data Validation & Frameworks
Great Expectations / Pandera: Proficiency in using Python-based libraries to define data contracts and automated validation suites.
DLT Expectations: Deep understanding of Delta Live Tables (DLT) expectations (Fail, Drop, Quarantining bad records).
Advanced SQL: Expert-level SQL for complex data reconciliation, identifying duplicates, and null-value analysis across billions of records.
Python for QA (PySpark):
Pytest-Spark: Experience using pytest to write unit tests for PySpark transformations and logic.
Notebook Testing: Ability to write automated test notebooks that validate Medallion Architecture transitions (Bronze to Silver, Silver to Gold).
Data Reconciliation: Building Python scripts to perform source-to-target counts and checksums across distributed file systems.
Performance & Integration Testing:
Scalability Testing: Ability to validate that data pipelines meet performance SLAs when data volume spikes.
End-to-End Orchestration Testing: Testing the reliability of Databricks Workflows and handling of job failures/retries.
Schema Evolution: Testing how pipelines handle upstream schema changes without breaking downstream Gold tables.
Governance & Security Testing:
Unity Catalog Validation: Testing Row-Level Security (RLS) and Column-Level Masking to ensure unauthorized users cannot see sensitive data.
Data Lineage: Validating that data lineage in Unity Catalog correctly reflects the movement of data across the Lakehouse.