Senior Data Engineer

Job Summary

We are seeking an experienced Senior Databricks Developer with 5 to 8 yrs of experience in designing and implementing scalable data engineering solutions. The ideal candidate will have strong expertise in big data processing, data pipeline development, and cloud-based data platforms, with hands-on experience using Databricks and distributed data processing frameworks like Apache Spark.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
• Build and optimise large-scale data processing solutions using Apache Spark.
• Develop and manage ETL/ELT workflows for ingesting, transforming, and processing structured and unstructured data.
• Write data processing scripts using Python, Scala, and SQL.
• Implement and manage data lake and data warehouse architectures.
• Optimise performance of data pipelines and Spark jobs.
• Integrate Databricks with cloud data services and storage platforms.
• Collaborate with data scientists, analysts, and engineering teams to deliver reliable data solutions.
• Implement best practices for data governance, data quality, and security.
• Troubleshoot data pipeline issues and ensure system reliability and scalability.
• Document data architecture, workflows, and operational processes.

Required Skills & Qualifications

• 5 to 8 years of experience in data engineering or big data development.
• Good hands-on experience with Databricks.
• Extensive experience with distributed data processing using Apache Spark.
• Proficiency in Python, Scala, and SQL.
• Strong experience building scalable ETL/ELT data pipelines.
• Experience working with large datasets in data lake or data warehouse environments.
• Experience with cloud platforms such as Microsoft Azure, Amazon Web Services, or Google Cloud Platform.
• Strong knowledge of data modelling, data architecture, and data integration concepts.
• Excellent analytical, troubleshooting, and performance optimisation skills.