Role Overview
The Lead Data Engineer will be responsible for designing, building, and optimizing scalable data pipelines. The role requires strong hands‑on experience in data engineering, distributed data processing.
Key Responsibilities
- Design, build, and maintain ETL/ELT pipelines.
- Develop and optimize data ingestion frameworks, data transformations, and end‑to‑end workflows for batch and streaming use cases.
- Work with stakeholders to understand data requirements and translate them into scalable data engineering solutions.
- Manage and optimize codes
- Ensure data quality, reliability, and observability through validation frameworks and monitoring.
- Contribute to data modelling, metadata management, and best practices within the data platform.
- Collaborate closely with data scientists, analysts, and business teams to support analytics and ML workloads.
Must have Skills & Experience
- 10+ years of experience in data engineering development
- Hands‑on experience with 5+ years in PySpark distributed data processing.
- 3+ years of indepth hands on experience in Python
- Atleast 3 + years of experience in Hadoop
- 3+ years of experience as a Data Engineer in AWS or Azure
- Solid SQL knowledge and experience working with large-scale datasets
- Good understanding of Delta Lake, medallion architecture, and scalable lakehouse patterns.
- Good understanding of CI/CD, Git, and modern DevOps practices for data pipelines.
- Familiarity with structured/unstructured data, data quality frameworks, and performance tuning.
