Data Engineer

Responsibilities

• Design and develop scalable backend services that integrate with external vendor systems through RESTful APIs
• Implement reliable data ingestion services, such as time-window retrieval and pagination
• Manage API error handling, retry mechanisms, and idempotent processing to ensure dependable data ingestion
• Build and maintain data pipelines to process high volumes of transactional and event data
• Carry out data transformation, validation, and enrichment for downstream systems
• Support both batch and near real-time data processing workflows
• Develop and deploy applications using Azure cloud services, such as Azure Functions, App Services, Storage, Key Vault, APIM, VNET, etc.
• Ensure solutions are built to be scalable, resilient, and highly available
• Implement observability practices, including logging, monitoring, and alerting
• Ensure data accuracy, consistency, and reconciliation across multiple data sources
• Support auditability and traceability of data flows to meet regulatory requirements
• Work with external vendors to define and refine API contracts and integration designs
• Deliver high-quality, maintainable, and reusable software solutions
• Contribute to team engineering standards and best practices
• Support continuous delivery through CI/CD pipelines and automated testing

Qualifications

• Strong proficiency in Python, preferred, or other backend languages such as Java or Node.js, along with strong proficiency in SQL, such as T-SQL, MySQL, or pgSQL
• Solid experience in developing backend services and API integrations
• Experience working with RESTful APIs, JSON payloads, and API consumption patterns
• Knowledge of secure integration patterns, such as API keys, TLS, certificates, and PGP
• Strong understanding of:
• Pagination, filtering, and incremental data retrieval
• Idempotency, retries, and failure handling in distributed systems

Required Skills

• Experience building data pipelines, either batch or streaming
• Familiarity with data transformation, validation, and data modelling
• Experience handling large datasets and high-volume data ingestion systems
• Understanding of data reconciliation and audit requirements will be an advantage
• Experience using data visualization tools such as Databricks Dashboard, Power BI, or Tableau

Preferred Skills

• Experience with relational databases such as PostgreSQL or MS SQL
• Knowledge of query optimisation and managing large datasets
• Experience with cloud platforms, preferably Microsoft Azure
• Experience setting up CI/CD pipelines and using GitLab and other relevant tools, such as GitLab CI/CD, SHIP, or HATS
• Familiarity with containerisation, such as Docker, will be an advantage
• Experience implementing:
• Logging and monitoring
• Metrics and alerting
• Understanding of system reliability and fault-tolerant design
• Experience integrating with third-party or vendor systems
• Experience working with event-driven systems or messaging platforms, such as Kafka or Event Hub
• Exposure to IoT, device data, or telemetry systems