Data Engineer

We are seeking a driven and technically sound Data Engineer to join our team in developing and maintaining enterprise data products for our organization's core Data Platform. In this role, you will build and optimize data pipelines handling vital operational datasets, including integrations with external data registries and multi-agency enterprise networks.

The ideal candidate will have at least 3 years of hands-on commercial experience building production-grade data workflows , implementing automated data-quality frameworks , and ensuring compliance with secure data-handling standards.

Key Responsibilities

1. Data Pipeline Engineering & API Integration

• Design, develop, and maintain automated batch and real-time ETL/ELT processes using cloud services (AWS Glue, Athena, S3, RDS) and modern databricks environments.

• Architect and deploy secure, cloud-native API integration frameworks leveraging serverless tools and data warehouses to ingest unstructured and structured external data sources with minimal retrieval latency.

• Utilize orchestration tools like Apache Airflow to manage workflow scheduling, dependency mappings, and automated batch execution.

2. Data Architecture, Modeling & Virtualization

• Implement scalable conceptual, logical, and physical data models utilizing Star/Snowflake schemas or dimensional modeling to establish robust analytical data marts.

• Work directly with data virtualization concepts using enterprise tools to build efficient virtual queries and analytical views that serve business intelligence and reporting dashboards.

• Help manage and sustain a common data model to consolidate transactional fields across data lakes, establishing a single source of truth.

3. Data Quality, Governance & Security Compliance

• Re-engineer dataset onboarding frameworks by building automated data validation and metadata verification checks to catch anomalies early and ensure high data integrity.

• Implement secure data-handling practices, including data masking, tokenization, anonymization, and role-based access control policies to safeguard sensitive records following least-privilege principles.

• Optimize database and query performance (including index management, table partitioning, and script refactoring) to minimize execution runtimes and compute costs.

4. Agile Collaboration & Documentation

• Work actively within an Agile/Scrum environment, participating in sprint planning, daily stand-ups, and retrospective ceremonies.

• Support automated code deployment and version control workflows utilizing continuous integration and Git/GitLab platforms.

• Create and maintain thorough technical documentation, including data dictionaries, data lineage flows, and architectural mapping to support long-term maintainability.

Qualifications and Experience

• At least 3 to 8 years of actual, post-graduate commercial work experience in data engineering or a highly related software development role.

• Strong, hands-on proficiency in Python, SQL, PySpark/Scala, and Bash scripting.

• Practical experience building data structures on cloud platforms, specifically AWS services (Glue, Athena, S3, RDS, Lambda) or related cloud-native environments.

• Foundational knowledge of data virtualization concepts familiarity with virtual query optimization is a major advantage.

• Experience working with version control ecosystems (Git/GitLab) and continuous integration/continuous deployment (CI/CD) pipelines.