Lead Data Engineer
- Posted 3 months ago
- Be among the first 10 applicants
Job Description
Position: Lead Data Engineer
Location: Bengaluru
Experience: 10+ years
Job Summary:
We are seeking an experienced Senior Data Engineer with a strong background in PySpark, Spark, and Big Data technologies. The ideal candidate will have over five years of hands-on experience designing, building, and optimizing large-scale data processing systems and pipelines.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines and ETL processes using PySpark and Spark.
- Perform data wrangling, transformation, and integration across diverse data sources.
- Optimize the performance and reliability of data processing workflows.
- Implement data quality frameworks to ensure data accuracy and integrity.
- Develop and maintain data architectures and data models to support business needs.
- Collaborate with cross-functional teams including data analysts, software engineers, and business stakeholders to understand data requirements.
- Stay updated with the latest advancements in Big Data tools, frameworks, and best practices.
- Mentor and guide junior data engineers in technical and professional growth.
Required Skills and Qualifications:
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
- 5+ years of experience in data engineering roles.
- Expertise in PySpark and Spark for large-scale data processing.
- Strong understanding of Big Data ecosystems and technologies (e.g., Hadoop, Hive).
- Hands-on experience with ETL development, data modeling, and data warehousing.
- Proficiency in programming languages such as Python, Scala, or Java.
- Experience with cloud platforms like AWS, Azure, or Google Cloud.
- Strong problem-solving skills and a detail-oriented mindset.
- Excellent communication and collaboration skills.
Preferred Qualifications:
- Experience with containerization and orchestration tools like Docker and Kubernetes.
- Knowledge of SQL and relational database management systems.
- Familiarity with data orchestration tools like Airflow, Prefect, or similar.
More Info
Key Skills
About Company
Skills:
Large Language Models (LLM), Pyspark, Azure Databricks, Sql, Rest Apis, Visual Studio Code, Python, Azure DevOps, Generative AI, Cloud-based data platforms, Scalable data architectures, ETL processes, Monitoring and alerting systems
Skills:
snowflake , Databricks, Sql, ELT, Data Lineage, Apache Spark, Data Lake, Pyspark, Goldengate, AWS, Etl, Data Quality, Python, Data Governance, Lakehouse, Observability, Data Pipelines, AWS DMS
Skills:
change data capture , Java, SQL Server, Spring Boot, Data Migration, Reconciliation, Messaging, relational database architecture, consistency models, SQL performance, distributed services, Transactions
Skills:
snowflake , Sql, ELT, S3, Lambda, Etl, Data Governance, Python, Iam, Cloudwatch, Monitoring, dbt, Secret Manager, Data quality checks, AWS cloud services, CI/CD pipelines
Skills:
data engineering , snowflake , Unix, Sql, ELT, Data Architecture, Qlik Replicate, Gitlab, Etl, AWS, Python, Snowpark, Data Design Patterns, dbt