What The Role Is
This role is responsible for data infrastructure and data pipeline development, collaborating with system architects or senior engineers to design and implement suitable data architecture, schema, and data storage solutions that support business requirements. The role works with vendors and stakeholders on secure API integrations, including API specifications and documentation, to integrate data from multiple sources, and creates end-to-end data pipelines by engaging with cross-functional teams such as analysts and data scientists to understand business requirements, ensuring data is properly formatted and accessible for analysis and reporting.
What You Will Be Working On
Data Infrastructure and Data Pipeline Development
Collaborate with system architect or senior engineers to design and implement the suitable data architecture, schema and data storage solutions that support business requirements. Work with vendor and stakeholders on secure API integrations, including API specification and documentations to integrate data from multiple sources. Create end-to-end data pipelines from various sources, engaging with cross-functional teams including analysts and data scientists to understand business requirements throughout the development process, ensuring it's properly formatted and accessible for analysis and reporting.
Performance Optimisation
Optimise system and pipeline performance through ongoing monitoring and implementation of scalable solutions designed to handle expanding data volumes and growing organisational demands.
Data Quality and Governance
Implement data quality checks and monitoring systems to ensure accuracy, completeness, and consistency of data at various stages of the data flow, ingestion and processing. Establish and maintain data governance practices and documentation. This includes troubleshooting issues when they arise.
Collaboration & Support
Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions that meet their needs. Provide technical support and guidance on data-related matters.
What We Are Looking For
- Demonstrated expertise in data engineering principles and scalable pipeline architecture.
- Required technical skills encompass advanced Python and SQL programming, cloud platform experience, and working knowledge of big data tools and frameworks.
- Familiar with cloud technologies such as AWS infrastructure and native AWS tools (CloudWatch and Glue).
- Familiar with data modelling, data access, and data storage infrastructure like Data Lake, Data Virtualisation and Data Warehouse for efficient storage and retrieval.
- Familiar with rest API and web requests/protocols in general.
- Capable of operating independently whilst collaborating effectively across diverse functional teams.
- Strong analytical thinking and problem-solving competencies.
- Excellent communication skills, both verbal and written.
- Minimum 2 years of relevant experience required. Fresh graduates are welcome to apply.
