We are seeking a Databricks Data Engineer to design, develop, and optimize modern data platforms using Databricks and cloud technologies. The successful candidate will have strong expertise in building scalable data pipelines, implementing Delta Lake architectures, developing PySpark-based transformations, and supporting enterprise analytics and AI initiatives. This role involves working closely with cross-functional teams to deliver high-quality, reliable, and governed data solutions.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks for enterprise data platforms.
- Build and optimize ETL/ELT processes using PySpark, Spark SQL, Databricks Notebooks, Workflows, and Delta Lake.
- Develop reliable data ingestion pipelines from relational databases, APIs, cloud storage, streaming platforms, log files, and external data sources.
- Design and implement Medallion Architecture (Bronze, Silver, Gold) data pipelines.
- Develop efficient data transformation, cleansing, enrichment, aggregation, and validation processes.
- Optimize Spark jobs, partitioning strategies, caching, and query performance for large-scale datasets.
- Implement data quality checks, monitoring, alerting, and automated recovery mechanisms.
- Build scalable batch and real-time data processing solutions.
- Work with Azure Data Factory (ADF), Airflow, or similar orchestration tools to automate data workflows.
- Collaborate with analytics, BI, machine learning, and infrastructure teams to deliver reliable data products.
- Implement data governance, security, lineage, and metadata management following enterprise standards.
- Participate in data modelling, architecture design, and platform modernization initiatives.
- Perform data profiling, reconciliation, SIT, UAT support, and production deployments.
- Contribute to CI/CD pipelines, Git-based version control, and infrastructure automation.
- Troubleshoot production issues and continuously improve platform performance and reliability.
- Evaluate new Databricks features and cloud-native technologies to enhance the data platform.
Requirements
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
- 3+ years of hands-on experience developing enterprise data solutions using Databricks.
- Strong experience with Databricks Lakehouse Platform, including:
- Delta Lake
- Databricks Notebooks
- Databricks Workflows
- Unity Catalog
- Delta Live Tables (DLT) (preferred)
- Auto Loader (preferred)
- Strong hands-on experience with PySpark and Spark SQL.
- Experience designing scalable ETL/ELT frameworks and distributed data processing solutions.
- Strong knowledge of data modelling, partitioning, optimization, and performance tuning.
- Experience building both batch and streaming data pipelines.
- Experience with Azure cloud services, including Azure Data Factory, Azure Data Lake Storage (ADLS), Azure Key Vault, and related Azure data services.
- Familiarity with streaming technologies such as Apache Kafka, Apache Flink, or Azure Event Hubs.
- Strong SQL skills, including complex queries, window functions, and query optimization.
- Experience using Git, Azure DevOps, and CI/CD pipelines.
- Understanding of Agile/Scrum development methodologies.
- Strong analytical, troubleshooting, and problem-solving skills.
- Excellent communication and collaboration skills.
- Experience with AWS data services is an added advantage.
Preferred Qualifications
- Databricks Certified Data Engineer Associate
- Databricks Certified Data Engineer Professional
- Experience implementing enterprise-scale Lakehouse architectures.
- Experience supporting AI, machine learning, or advanced analytics workloads using Databricks.
- Knowledge of data governance, metadata management, and data security best practices within Databricks environments.
