Senior Data Engineer

We are looking for an experienced Senior Data Engineer with 8-12 years of experience in Data Engineering, Big Data, Data Lake, and Lakehouse platforms. The ideal candidate will have strong hands-on expertise in Databricks, Spark, Pyspark, Python, SQL, cloud data platforms, data products, multimodal data pipelines, and modern data architectures.

Key Responsibilities

• Implement and operationalize enterprise Lakehouse platforms, Data Products, and Data Marketplace capabilities.
• Design and develop scalable batch, streaming, CDC, and API-based data ingestion pipelines.
• Build multimodal data ingestion pipelines supporting
• Build, test, and maintain foundation and business data products with defined data contracts, SLAs, and data quality controls.
• Implement and work with open table formats such as Apache Iceberg, Apache Hudi, and Delta Lake.
• Develop and support data pipelines for RAG, vector search, Generative AI, and agentic AI use cases.
• Perform performance tuning, optimization, production support, troubleshooting, and root cause analysis.
• Develop technical documentation, deployment guides, operational runbooks, and support procedures.
• Ensure compliance with engineering standards, DevSecOps controls, CI/CD practices, and software delivery standards.
• Collaborate with distributed engineering, architecture, data science, and business teams across multiple projects.

Key Technical Requirements

• 8-12 years of experience in Data Engineering, Big Data, Data Lake, or Lakehouse implementations.
• Hands-on experience with Databricks, Snowflake, Cloudera, and cloud data platforms such as Azure, AWS, or GCP.
• Strong experience building Data Products and Data Marketplace capabilities.
• Excellent programming skills in Python, Scala, Java, and SQL.
• Strong hands-on expertise with Spark / PySpark.
• Experience with Iceberg, Hudi, Delta Lake, and object storage platforms.
• Strong experience with data ingestion, transformation, reconciliation, and data quality frameworks.
• Experience with Kafka, Flink, Spark Streaming, Airflow, Trino, Dremio, Hive, and Impala.
• Hands-on experience with Kubernetes, OpenShift, Docker, Terraform, Jenkins, Git, and CI/CD pipelines.
• Experience with MLflow and observability platforms/tools.
• Ability to design data architectures for NLP, AI/ML, GenAI, and unstructured data.
• Experience with ML platforms and libraries such as CML, Spark MLlib, scikit-learn, and XGBoost, including model deployment.
• Experience developing internal engineering tools and applications using Python, Shell scripting, Flask, React, or similar modern frameworks.
• Knowledge of data modeling, metadata management, data lineage, governance, APIs, event streams, dashboards, and BI platforms.
• Experience with Teradata, Netezza, Greenplum, or MPP migration programs is an advantage.