Role Summary
We are seeking a highly skilled Big Data Engineer with strong experience in PySpark and Scala to design, build, and optimize large-scale data processing systems. The candidate will work on distributed data platforms, enabling real-time and batch analytics solutions across enterprise data ecosystems.
The role involves building scalable data pipelines, data lakes, and streaming solutions leveraging modern big data technologies.
Key Responsibilities
- Data Engineering & Development
- Design and develop scalable data pipelines using PySpark and Scala
- Data Platform & Architecture
- Data Processing & Optimization
- Streaming & Real-Time Data
- Cloud & Modern Data Stack
- Data Quality & Governance
- Collaboration & Delivery
- Required Skills
- Core Technologies
- Strong hands-on experience in:
- PySpark & Apache Spark
- Scala programming
- Expertise in:
- Spark SQL, DataFrames, RDDs
- Distributed data processing frameworks
- Big Data Ecosystem
- Hadoop: HDFS, Hive, YARN
- Data ingestion tools: Kafka, Sqoop
- Data formats: Parquet, ORC, JSON, Avro
- Programming & Query
- Python / Scala
- Advanced SQL (joins, aggregations, optimization)
- Shell scripting / Unix basics
- Streaming & Messaging
- Kafka (must-have)
- Event-driven architecture
- Real-time data processing frameworks
- DevOps & Tools
- Git, CI/CD pipelines
- Docker / Kubernetes (good to have)
- Scheduling tools (Airflow / Control-M)
- Cloud (One or more preferred)
- AWS / Azure / GCP
- Databricks / Snowflake exposure
- ibu
- itcan
- Domain Experience (Preferred)
- BFSI / Banking / Payments domain
- Experience working with high-volume financial datasets
- Knowledge of data compliance and regulatory reporting
- This role requires a combination of:
- Strong PySpark + Scala coding
- Deep big data architecture knowledge
- Hands-on experience with streaming + cloud data engineering
