Company Overview
Infogain Solutions is a leading IT services company specializing in IT solutions, staffing, and recruitment across industries including financial services, IT, and audit firms. We connect professionals with organizations to drive business growth and innovation.
Job Summary
As a Data Engineer, you will design and implement scalable data pipelines and infrastructure using Spark, Hadoop, and container platforms to support large-scale data processing and analytics.
Responsibilities
- Design, develop, and implement Spark Scala applications and data pipelines to process large volumes of structured and unstructured data efficiently
- Integrate Elasticsearch with Spark to enable effective data indexing, querying, and retrieval
- Optimize Spark jobs for performance and scalability to ensure efficient data processing and indexing in Elasticsearch
- Collaborate with data engineers, data scientists, and stakeholders to translate requirements into technical specifications and solutions
- Design and deploy data engineering solutions on OpenShift Container Platform using containerization and orchestration techniques
- Optimize containerized data workflows for efficient resource utilization and deployment
- Work with DevOps teams to streamline deployment processes, implement CI/CD pipelines, and maintain platform stability
- Monitor and troubleshoot data pipeline performance issues, implementing enhancements to improve reliability and efficiency
- Implement monitoring and logging mechanisms to ensure data infrastructure health, availability, and performance
- Document data engineering processes, workflows, and infrastructure configurations for knowledge sharing and reference
- Stay current with emerging technologies, industry trends, and best practices in data engineering and DevOps
- Provide technical leadership and mentorship to junior team members to foster continuous learning and innovation
Required competencies and certifications
- Bachelor's degree in Computer Science, Data Engineering, Information Technology, or related field
- Minimum 10 years of experience as a Data Engineer working with Hadoop, Spark, and large-scale data processing technologies
- Expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Ranger)
- Experience with containerization platforms such as OpenShift Container Platform and Kubernetes orchestration
- Proficiency in programming languages used in data engineering such as Spark, Python, Scala, or Java
- Knowledge of DevOps practices, CI/CD pipelines, and infrastructure automation tools including Docker, Jenkins, Ansible, and BitBucket
- Strong problem-solving and troubleshooting skills with a proactive approach to resolving technical challenges
- Excellent collaboration and communication skills to work effectively with cross-functional teams
- Ability to manage multiple priorities, meet deadlines, and deliver high-quality results in a fast-paced environment
Preferred competencies and qualifications
- Experience with monitoring tools such as Grafana, Prometheus, and Splunk
- Experience with cloud platforms and their data services (e.g., AWS, Azure, GCP)
