Senior Data Engineer Databricks

Job Description

. We are looking for an experienced Senior Data Engineer - Databricks to design, develop, and maintain scalable data pipelines on the Databricks platform.

. The role requires strong expertise in PySpark, Databricks, and modern data engineering practices, along with experience modernizing legacy data pipelines and building high-performance data solutions in cloud environments.

Key Responsibilities

. Design, build, and maintain scalable data pipelines using Databricks and PySpark.

. Develop end-to-end data workflows including data ingestion, transformation, and consumption.

. Optimize Spark jobs, cluster configurations, and pipeline performance.

. Manage and orchestrate workflows using Databricks Jobs and notebooks.

. Refactor legacy ETL pipelines into modern PySpark-based ELT frameworks.

. Implement data quality checks, monitoring, and error handling mechanisms.

. Design and maintain Delta Lake tables with proper optimization strategies.

. Collaborate with data architects, analysts, and infrastructure teams to deliver reliable data solutions.

. Troubleshoot and resolve production data pipeline issues.

Required Skills & Qualification

. Strong Data Engineering fundamentals including ETL/ELT pipeline design.

. Hands-on experience with PySpark (DataFrames API, Spark SQL, performance tuning).

. Experience with Databricks platform including workspace, clusters, notebooks, and job orchestration.

. Knowledge of Delta Lake features such as ACID transactions and schema evolution.

. Strong Python programming for data processing and automation.

. Experience with data modelling (Dimensional modeling, Data Vault, or Lakehouse architecture).

. Strong SQL skills for data transformation and analysis.

. Experience with cloud platforms (Azure, AWS, or GCP).

. Experience with Git version control and CI/CD practices.

. Minimum 8 years of experience in data engineering or related roles.

. 2-3 years of hands-on experience with Databricks.

. Experience building and maintaining production-grade data pipelines at scale.

. Proven experience modernizing legacy data pipelines.

. Experience working in Agile development environments.

. Certifications (Mandatory) Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional