You will be responsible for operating and evolving SP Group's enterprise Data & AI platform
to support mission-critical analytics, regulatory compliance, and AI-driven innovation. This
role focuses on ensuring the platform is scalable, reliable, secure, and performant, while
enabling data engineers, scientist and analysts to deliver business value efficiently.
What You'll Do
- Ensure high availability and reliability of the enterprise data platform (Cloudera and Talend Big Data).
- Deliver scalable and performant data infrastructure supporting batch and near real-time workloads.
- Enable secure and compliant data operations aligned with enterprise and regulatory requirements.
- Improve platform automation and operational efficiency through DevOps and Infrastructure-as-Code.
- Provide strong platform support and partnership to data engineering, analytics, and business teams.
Data Platform Engineering & Operations
- Design, deploy, and manage large-scale data platform environments (primarily Cloudera and Talend Big Data).
- Operate and optimize Hadoop ecosystem components (HDFS, Hive, Spark, HBase, Impala, Iceberg, Ozone).
- Manage cluster lifecycle including provisioning, scaling, patching, and upgrades.
- Ensure platform uptime, stability, and service reliability through proactive monitoring and maintenance.
Performance, Scalability & Capacity Management
- Monitor workload performance and optimize data processing efficiency.
- Perform capacity planning to support increasing data volumes and usage.
- Troubleshoot performance bottlenecks across compute, storage, and query layers.
Platform Delivery & Execution Planning
- Plan and coordinate execution of platform upgrades, patching, and scaling activities.
- Manage deployment schedules and ensure minimal service disruption.
- Coordinate cross-team dependencies (infrastructure, network, security).
- Track and report execution progress against platform roadmap.
DevOps, Automation & Platform Engineering
- Implement Infrastructure-as-Code using tools such as Terraform and Ansible.
- Design and maintain CI/CD pipelines for platform configuration.
- Automate platform provisioning, deployment, and operational workflows.
- Improve observability through monitoring, alerting, and logging practices.
Security, Governance & Compliance
- Implement and manage platform security controls including authentication (Kerberos), authorization (e.g. Ranger), and encryption.
- Ensure compliance with enterprise data governance, security policies, and regulatory requirements.
- Support auditability, access control, and data protection mechanisms.
Collaboration & Platform Enablement
- Work closely with data engineers, data scientists, and business teams to enable data solution delivery.
- Partner with infrastructure, network, and security teams to ensure platform integration and stability.
- Provide guidance, troubleshooting, and best practices to users of the platform.
- Document platform architecture, processes, and operational procedures.
What You'll Need
- 5- 10 years of experience in data platform engineering, big data operations, or infrastructure engineering.
- Strong hands-on experience managing big data platforms (Cloudera CDP, Hadoop ecosystem).
- Deep understanding of Hadoop components: HDFS, Hive, Spark, HBase, and related services.
- Experience with cluster operations, performance tuning, and troubleshooting.
- Proficiency in automation and scripting (Python, Bash, Terraform, Ansible).
- Experience with CI/CD pipelines and DevOps practices.
- Solid understanding of data platform security and access control concepts.
- Strong analytical and problem-solving skills with a proactive mindset.
- Ability to work in a fast-paced, cross-functional environment.
- Good communication skills with ability to translate technical topics to stakeholders.
- Ownership mentality with focus on platform reliability and continuous improvement.
Good to have
- Experience with Databricks, Talend, or hybrid data platforms.
- Knowledge of containerization and orchestration.
- Experience with streaming platforms (e.g. Kafka) or real-time processing.
- Familiarity with modern data lake-house concepts (e.g. Iceberg, Delta).
- Exposure to cloud or hybrid environments (Azure preferred).
What We'll Provide
- Join us to empower the future of energy .
- You will work in teams that thrive on collaboration for outstanding achievements.
- We have regular wellness programmes to promote healthy living
