We are looking for an experienced and highly skilled Hadoop Data Engineer to join our dynamic team. The ideal candidate will have hands-on expertise in developing optimized data pipelines using Python, PySpark, Scala, Spark-SQL, Hive, and other big data technologies. You will be responsible for translating complex business and technical requirements into efficient data pipelines and ensuring high-quality code delivery through collaboration and code reviews.
Roles & Responsibilities:
Data Transformation & Pipeline Development:
- Design and implement optimized data pipelines using PySpark, Python, Scala, and Spark-SQL.
- Build complex data transformation logic and ensure data ingestion from source systems to Data Lakes (Hive, HBase, Parquet).
- Produce unit tests for Spark transformations and helper methods.
Collaboration & Communication:
- Work closely with Business Analysts to review test results and obtain sign-offs.
- Prepare comprehensive design and operational documentation for future reference.
Code Quality & Review:
- Conduct peer code reviews and act as a gatekeeper for quality checks.
- Ensure quality and efficiency in the delivery of code through pair programming and collaboration.
Production Deployment:
- Ensure smooth production deployments and perform post-deployment verification.
Technical Expertise:
- Provide hands-on coding and support in a highly collaborative environment.
- Contribute to development, automation, and continuous improvement practices.
System Knowledge:
- Strong understanding of data structures, data manipulation, distributed processing, and application development.
- Exposure to technologies like Kafka, Spark Streaming, and ML is a plus.
RDBMS & Database Management:
- Hands-on experience with RDBMS technologies (MariaDB, SQL Server, MySQL, Oracle).
- Knowledge of PLSQL and stored procedures is an added advantage.
Other Responsibilities:
- Exposure to TWS jobs for scheduling.
- Knowledge and experience in Hadoop tech stack, Cloudera Distribution, and CI/CD pipelines using Git, Jenkins.
- Experience with Agile Methodologies and DevOps practices.
Technical Requirements:
- Experience: 6-9.5 years of experience in Hadoop, Spark, PySpark, Scala, Hive, Spark-SQL, Python, Impala, CI/CD, and Git.
- Strong understanding of Data Warehousing Methodology and Change Data Capture (CDC).
- In-depth knowledge of Hadoop & Spark ecosystems with hands-on experience in PySpark and Hadoop technologies.
- Proficiency in working with RDBMS such as MariaDB, SQL Server, MySQL, or Oracle.
- Experience with stored procedures and TWS job scheduling.
- Solid experience with Enterprise Data Architectures and Data Models.
- Background in Core Banking or Finance domains is preferred experience in AML (Anti-Money Laundering) domain is a plus.
Skills & Qualifications:
- Strong hands-on coding skills in Python, PySpark, Scala, Spark-SQL.
- Proficient in Hadoop ecosystem (Hive, HBase, etc.).
- Knowledge of CI/CD, Agile, and DevOps methodologies.
- Good understanding of data integration, data pipelines, and distributed data systems.
- Experience with Oracle, PLSQL, and working with large-scale databases.
- Strong analytical and problem-solving skills, with an ability to troubleshoot complex data issues.
