Senior Data Engineer
The Senior Data Engineer plays a critical role in designing, developing, and maintaining scalable data infrastructure and pipelines. This position supports data-driven decision-making across the organization by ensuring data quality, accessibility, and performance.
Roles and Responsibilities
- Data Pipeline Development:Design, build, and maintain robust, scalable data pipelines using tools like Apache Airflow, Kafka, and Spark.
- Data Architecture:Develop and optimize data models, data warehouses, and ETL processes to support analytics and machine learning initiatives.
- Collaboration:Work closely with data scientists, analysts, and software engineers to understand data requirements and deliver tailored solutions.
- Data Quality & Governance:Ensure data integrity, implement validation checks, and enforce data governance policies.
- Performance Optimization:Monitor and improve data processing speed, system uptime, and query performance.
- Leadership & Mentorship:Guide junior engineers, promote best practices, and foster a culture of continuous learning.
- Innovation & Research:Stay updated with emerging technologies in AI, automation, and data streaming; lead initiatives to integrate them into existing systems.
- Other duties as required.
Desired Skills and Experience
- Bachelor's or master's degree in computer science, data science, information technology, or a related field, with 6+ years of experience.
- Proficiency inSQL,Python, andETL/ELT tools
Experience in Microsoft solutions are as follows:
o Azure Data Factory
o Microsoft Fabric
o Azure Synapse Analytics
o Azure Databricks
o Power BI
o Azure SQL Database / SQL Server
o Azure Purview
- Experience withbig data technologies(Hadoop, Spark)
- Familiarity withcloud platforms(Azure, AWS, and GCP)
- Strong understanding ofdata modelling,data warehousing, andstream processing
- Knowledge ofdata security,compliance, andgovernance
- Knowledge of CI/CD pipelines using Azure DevOps
- Understanding of data security and compliance (e.g., GDPR, HIPAA)
