Role Overview
We are seeking an experienced Senior Data Engineer to design, develop, and maintain scalable big data pipelines supporting advanced analytics and data intelligence initiatives. The role will focus on building high performance data platforms and analytics capabilities using the Quantexa platform within a large-scale enterprise data environment.
Key Responsibilities
- Design, develop, and maintain large-scale data pipelines using Hadoop and Spark.
- Implement data transformation, aggregation, and enrichment pipelines for advanced analytics.
- Deploy and optimize data workloads on OpenShift Container Platform (OCP).
- Collaborate with DevOps teams to implement CI/CD pipelines and automation frameworks.
- Ensure data governance, lineage tracking, metadata management, and compliance standards.
- Monitor and optimize the performance of distributed data processing systems.
- Implement logging, monitoring, and alerting mechanisms using Grafana, Prometheus, and Splunk.
- Provide technical leadership and mentorship to junior engineers.
- Work closely with cross-functional stakeholders to translate business and analytics requirements into scalable data engineering solutions.
Key Requirements
- Bachelor's degree in Computer Science, Information Technology, or related discipline.
- Minimum 6+ years of experience in Data Engineering within large enterprise environments.
- Strong hands-on expertise in the Hadoop ecosystem (HDFS, Hive, Spark, Cloudera).
- Proficiency in Scala, Python, Java, and advanced SQL.
- Experience with OpenShift (OCP) and Kubernetes-based container orchestration.
- Strong knowledge of DevOps practices, CI/CD pipelines, Docker, Jenkins, and Ansible.
- Experience with enterprise job schedulers such as Control-M.
- Experience working with data analytics platforms supporting financial crime detection or entity resolution is preferred.
- Prior experience implementing or supporting the Quantexa platform is must.
- Exposure to cloud platforms (AWS, Azure, or GCP) is a plus.
