Senior Data Engineer

Role: Senior Data Engineer

Job Description:

The ideal candidate will play a key role in designing, developing, and maintaining scalable big data solutions while providing technical leadership. This role will also support strategic Data Governance initiatives, ensuring data integrity, privacy, and accessibility across the organisation.

Key Responsibilities:

• Design, implement, and optimise robust data pipelines and ETL/ELT workflows using SQL and Python.
• Lead architecture discussions, including the design and review of ERDs, as well as overall data modelling and system design.
• Collaborate closely with Data Engineers, Data Analysts, and cross-functional engineering teams to meet evolving data needs.
• Build and maintain synchronous and asynchronous data APIs to allow downstream systems to efficiently access and consume data.
• Lead the development and optimisation of Customer Data Platform (CDP) capabilities to enhance data integration, accessibility, and insights across the organisation.
• Deploy and manage cloud infrastructure using Terraform and other Infrastructure-as-Code (IaC) tools to ensure scalable, reliable, and well-structured environments.
• Develop and maintain CI/CD pipelines for deploying data applications and services.
• Leverage strong experience in AWS services (e.g., S3, Glue, Lambda, RDS, Lake Formation) to support scalable and secure cloud-based data platforms.
• Handle both batch and real-time data processing effectively.
• Apply best practices to support data governance, privacy, and security initiatives.
• Implement and manage data encryption and hashing techniques to secure sensitive information.
• Ensure adherence to software engineering best practices including version control, automated testing, and deployment standards.
• Lead performance tuning and troubleshooting for data applications and platforms.

Required Skills & Experience:

Strong proficiency in SQL for querying and transformation.

Advanced Python development skills with an emphasis on data engineering use cases.

Hands-on experience with Terraform for cloud infrastructure provisioning.

Proficiency with CI/CD tools, particularly GitHub Actions.

Deep expertise in AWS cloud architecture and services.

Demonstrated ability to create and evaluate architecture and documentation while contributing to architectural decisions.

Strong communication and leadership skills, with experience mentoring engineering teams.

Experience with big data technologies such as Apache Spark, Hive, or Kafka.

Familiarity with containerisation tools (e.g., Docker) and orchestration platforms (e.g., Kubernetes).

Solid understanding of data governance, data quality, privacy, and security frameworks.

---------------------------

Please refer to U3's Privacy Notice for Job Applicants/Seekers at https://u3infotech.com/privacy-notice-job-applicants/. When you apply, you voluntarily consent to the collection, use and disclosure of your personal data for recruitment/employment and related purposes.