Design, develop, and maintain robust ETL/ELT pipelines to process large volumes of structured data
Implement batch data processing solutions using Databricks
Monitor data pipeline performance and troubleshoot issues to ensure reliable data flow
Optimize existing pipelines for improved performance and cost efficiency
Data Architecture & Infrastructure
Collaborate with senior engineers to design and implement scalable data architectures
Build and maintain data warehouses, data lakes, and cloud-based storage solutions
Ensure data quality, consistency, and reliability across all systems and datamarts
Implement data governance best practices and security protocols defined by MOH HIM policies and DGPO
Technical Implementation
Develop and maintain notebooks for data access and integration
Write clean, efficient, and well-documented code following best practices
Implement automated testing and deployment processes for data pipelines
Collaboration & Communication
Partner with data scientists, analysts, clinical and business stakeholders to understand data requirements
Provide technical guidance on data availability, limitations, and best practices to stakeholders
Document data transformation processes, onboarding processes and automation processes
Participate in code reviews and contribute to team knowledge sharing
Requirements
Bachelor's degree in Computer Science, Information Systems, Analytics, or a relevant field of study
1-3 years of experience in data engineering or a related role
Proficient in SQL and at least one programming language (Python, Java, Scala)
Decent understanding of data warehousing concepts and ETL processes
Familiarity with cloud platforms such as AWS and Databricks
Familiarity with Git commands and Unix/Linux system experience
Experience with data visualization tools (Tableau, Power BI)
Strong analytical and problem-solving skills
Familiar with database management, architecture design, system integration, security and data governance
Good interpersonal skills, a detail-oriented and flexible person who can work across different areas within the team as well as align cross functionally outside the team
Ability to deliver clear, concise reports and presentations/dashboards and effectively articulate observations and recommendations
