We are seeking an experienced Data Engineer with excellent expertise in big data platforms, containerized environments, and DevOps practices. In this role, you will design, build, and maintain scalable data pipelines that process large-scale datasets supporting advanced analytics and machine learning solutions.

You will primarily work on the Quantexa platform within the AML domain, ensuring high availability, performance, and reliability of data infrastructure deployed on OpenShift Container Platform (OCP).

Key Responsibilities

Data Engineering & Analytics

Design, implement, and maintain data transformation, aggregation, and enrichment pipelines to support analytics and machine learning initiatives.
Process and manage large-scale datasets using Hadoop, Spark, and related technologies.
Ensure data quality, integrity, and consistency across the entire data lifecycle.
Implement data governance practices, including data lineage, metadata management, and compliance controls.

Platform & Containerization

Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques.
Optimize data workflows for containerized deployment and efficient resource utilization.
Monitor, tune, and enhance data pipeline performance to ensure scalability and reliability.

DevOps & Operations

Collaborate with DevOps teams to streamline deployments and implement CI/CD pipelines.
Implement monitoring and logging solutions to ensure health, availability, and performance of the data infrastructure.
Troubleshoot production issues and perform root cause analysis with a proactive approach.

Collaboration & Leadership

Work closely with cross-functional teams to translate data requirements into effective engineering solutions.
Document data pipelines, workflows, and infrastructure configurations for knowledge sharing.
Stay current with emerging technologies, industry trends, and best practices in data engineering and DevOps.
Provide technical guidance and mentorship to junior team members, contributing to continuous improvement of analytics capabilities.

Technical & Professional Requirements

Core Experience

Bachelor's degree in Computer Science, Information Technology, or a related discipline.
Minimum 6 years of experience as a Data Engineer in large-scale data environments.
Hands-on experience developing or maintaining Quantexa solutions (AML domain experience).

Big Data & Processing

Solid expertise in Hadoop ecosystem tools (HDFS, Hive, Pig, Cloudera).
Advanced experience with Apache Spark for large-scale data processing.
Proficiency in SQL and data modeling concepts.

Programming & Scripting

Proficiency in one or more of the following: Scala, Python, Java.
Working knowledge of Shell scripting.
Containerization, DevOps & Tooling
Hands-on experience with OpenShift Container Platform (OCP) and Kubernetes.
Good understanding of DevOps practices, CI/CD pipelines, and automation tools such as Docker, Jenkins, Ansible, and Bitbucket.
Experience with enterprise job schedulers such as Control-M.

Monitoring & Observability

Experience with monitoring and logging tools such as Grafana, Prometheus, and Splunk.
Cloud & Platform Exposure
Exposure to cloud platforms such as AWS, Azure, or GCP, including data services.
Quantexa exposure and/or certification.
EA License : 02C3423
EA Personnel : R22108699

Data Engineer- Quantexa