Data Engineer

1) Data Architecture & Engineering

Develop and maintain ETL/ELT pipelines that efficiently process large volumes of structured and unstructured data from diverse sources.

Ensure data quality, governance, and security standards are embedded throughout all data engineering processes.

2) Technical Implementation

Hands-on development of notebooks using PySpark, Python, and SQL for ETL automation.

Create and optimize PySpark scripts for efficient data extraction, transformation, and loading from large datasets.

Implement custom data manipulation, validation, and error handling solutions to enhance ETL robustness.

3) Performance & Optimization

Monitor and optimize data pipeline performance, implementing solutions for scalability and cost-effectiveness.

Conduct testing, debugging, and troubleshooting of data transformation processes.

Verify data integrity throughout pipeline stages and resolve complex technical issues.

Requirements

Minimum 3 years of data engineering experience with Azure-native implementations

Proven experience working as part of a data engineering team.

Hands on experience building pipeline using ADF, Synapse and Delta lake.

Demonstrated expertise developing PySpark scripts for efficient data extraction, transformation, and loading from large datasets

Experience utilizing Python for custom data manipulation, validation, and error handling in ETL processes

Proficiency employing SQL for complex joins, aggregations, and database operations within Databricks environments

Track record of testing, debugging, and optimizing data transformation processes for accuracy and performance

Experience verifying data integrity throughout pipeline stages and resolving troubleshooting issues