We are looking for a
Python Developer (Data Engineering) to design, build, and support scalable data pipelines and analytics platforms within a large enterprise banking environment. The role involves end-to-end data engineering across batch and near-real-time processing, strong collaboration with business and IT stakeholders, and adherence to high standards of quality, governance, and security.
Key Responsibilities
- Develop robust data ingestion, transformation, and loading (ETL/ELT) processes across batch and near-real-time workflows
- Implement distributed data processing using Apache Spark (PySpark / Scala) for large-scale transformations and analytics
- Design and maintain logical and physical data models (dimensional/star schemas, data vault, wide tables) optimized for reporting and analytics
- Write and optimize SQL and HiveQL queries; manage tables, partitions, and storage formats
- Schedule, monitor, and support data pipelines using Control-M, ensuring SLA adherence and timely delivery
- Perform performance tuning of Spark jobs, SQL/Hive queries, and storage strategies for scalability and cost efficiency
- Implement data validation, reconciliation, and lineage using checks, unit tests, and metadata frameworks
- Build operational dashboards and alerts, investigate failures, and drive root-cause analysis and remediation
- Maintain comprehensive runbooks, architecture diagrams, data dictionaries, and coding standards
- Apply data privacy, access control, and security best practices in line with enterprise and regulatory requirements
- Drive continuous service and process improvements
- Prepare unit test cases and work closely with Testing teams during SIT and UAT
- Package and migrate code across environments (DEV → QA → PROD) with proper audit trails and governance
