Big Data Cloud Engineer (HDFS, YARN, Kafka, Linux, AWS, Shell scripting)

Key Responsibilities

• Administer, maintain, and support large-scale Hadoop and Cloudera (CDH/CDP) environments across distributed enterprise infrastructures.
• Design, deploy, and manage Hadoop cluster architecture, ensuring high availability and scalability in production environments.
• Manage and optimize Big Data ecosystem components such as HDFS, YARN, Spark, Kafka, Hive, Impala, HBase, Oozie, and Zookeeper.
• Perform cluster monitoring, troubleshooting, and performance tuning to ensure system stability and efficiency.
• Support both real-time and batch data processing platforms using Spark, Kafka Streams, Hive, and HBase.
• Configure and maintain ecosystem tools including Hue, Airflow, NiFi, Sqoop, Datameer, Splunk, and Arcadia.
• Implement and manage security frameworks such as Kerberos, Ranger, Sentry, SSL, and IAM policies.
• Manage AWS cloud services including EC2, S3, EMR, IAM, and VPC for Big Data workloads.
• Conduct capacity planning, cluster expansion, and resource optimization to improve performance.
• Develop automation scripts using Shell scripting, Python, and PySpark to enhance operational efficiency.
• Perform system upgrades, patching, backup, and maintenance activities with minimal downtime.
• Monitor production systems using tools such as New Relic and other observability platforms.
• Lead incident management, root cause analysis (RCA), and production support for critical systems.
• Collaborate with DevOps, infrastructure, and application teams to support data engineering initiatives.
• Participate in Agile processes and continuous improvement initiatives.
• Mentor junior team members and promote best practices.
• Support enterprise platforms in banking/financial services environments with strict compliance requirements.
• Provide support for PeopleSoft administration and integration environments where required.

Requirements

• Minimum 10 years of experience in Hadoop and Cloudera (CDH/CDP) administration.
• Proven experience in the banking or financial services sector.
• Strong expertise in Hadoop cluster architecture, deployment, and performance tuning.
• Hands-on experience with HDFS, YARN, Kafka, Spark, Hive, Impala, MapReduce, HBase, Oozie, and Zookeeper.
• Strong background in Linux administration and infrastructure troubleshooting.
• Experience with AWS cloud services (EC2, S3, EMR, IAM, VPC).
• Proficiency in Shell scripting, Python, and PySpark for automation.
• Experience with PeopleSoft administration and Oracle GoldenGate is an advantage.
• Solid understanding of data security frameworks, including Kerberos, Ranger, and Sentry.
• Experience in monitoring, incident management, and performance tuning (e.g., New Relic).
• Experience supporting large-scale, high-availability environments.
• Familiarity with CI/CD pipelines, DevOps practices, and tools such as GitHub and Jenkins.
• Exposure to Big Data tools such as Hue, Airflow, NiFi, Sqoop, Datameer, Splunk, and Arcadia.
• Strong analytical, problem-solving, and stakeholder management skills.
• Excellent communication and teamwork abilities.