Ops Support Engineer

Role Developer / Applications Programmer

Skill Category Application Development - Scripting

Roles and Responsibilities:

EDR Server Operations & Maintenance

Maintain and operate Carbon Black (CB) EDR servers hosted on RHEL 8.

Perform scheduled maintenance tasks on Operating System (OS), databases and application (CB) level, including patching, health checks, backups, and service restarts under approved change windows.

Validate and maintain EDR server application services (web console, CB backend services, Solr, PostgreSQL) and log forwarding services (NXLog, CB event forwarder) to the PUB's Log Collector.

Monitor and clear stale services, queues, or indexing issues.

Manual review of system, security and audit logs of OS, database and application which are not forwarded to the Board's designated plant log collection server

Engage OEM to provide troubleshooting solution under software licence support, provide professional services for both hardware and software related configuration, integration and troubleshooting works if necessary

Endpoint Sensor Management

Monitor presence, health, and connectivity of all sensors deployed on monitored endpoints.

Troubleshoot sensors that are showing offline on the server:

o Validate network paths of sensor communication.

o Inspect sensor local service state, tamper protection status, and logs.

o Collect endpoint diagnostic data.

Support redeployment or recovery of corrupted/missing sensor components.

Assist in onboarding new hosts into the EDR environment.

Validate EDR server sensor information against plant asset inventory.

Network & Infrastructure Coordination

Work with appropriate third parties to validate port connectivity, firewall rules, and Carbon Black SSL certificate issues for endpoint integration.

Assist in troubleshooting endpoints inside air-gapped networks using host mappings and static resolving techniques.

Support analysis of intermittent or unstable site connectivity.

Security & Hardening

Ensure EDR server OS hardening (RHEL CIS baseline), account permissions, and file system access are maintained according to PUB Hardening Guides.

Monitor release of new security patches for OS and application of EDR system, track security vulnerabilities, assesses applicability and implement patches or workarounds according to PUB stipulated timeframe for patching. These changes shall be logged via change requests.

Quarterly review of User Access for the EDR servers, including but not limited to user/service/privileged accounts and access rights, user activities logs like successful login/logout events and failed login attempts, and inactive user accounts for more than 90 days.

Annual review of system configuration, including unnecessary services and applications, improper user account and password settings, improper logging and backup settings and improper windows/network security policy setting. For any configuration weaknesses identified, provide the Board with full details of the actions to be taken to harden or correct the weaknesses and perform risk assessments for hardening that cannot be applied.

Put up reviews in writing for approval by designated Approving Officer of the Board.

Maintain a key management system to track and manage the lifecycle of cryptographic keys used in the EDR system

Comply with any written instructions on cybersecurity related matters that are issued by the Government and Board from time to time.

Technical Support & Incident Response

Provide technical support for issues escalated by the cybersecurity branch.

Assist during cybersecurity investigations by retrieving endpoint data, event logs, and sensor telemetry.

Support forensic or operational queries where EDR data is required.

Reporting & Documentation

Produce maintenance reports after every maintenance cycle. The report shall minimally include:

o Summary status report of completed jobs, ad-hoc support and outstanding jobs;

o Server health checklist;

o Performance measurement (including System Availability, Response Time, Turnaround Time);

o System, security and audit log review and findings;

o Software security patch;

o Tracking of software license subscription expiry;

o Action item on outstanding matters with the Board

Maintain SOPs, inventory of assets, system configuration notes, and troubleshooting guides.

Raise change request according to PUB Change Management for maintenance works and system changes.

Document system changes, sensor onboarding actions, and backup logs.

Mandatory Technical Skills

Experience with Carbon Black EDR, or equivalent EDR platforms.

Strong RHEL system administration knowledge (RHEL 8/9).

Familiarity with PostgreSQL maintenance tasks (backup, check, restore).

Understanding of network troubleshooting (TCP, TLS, proxies, firewalls).

Hands-on experience with endpoint diagnostics for EDR sensors on Windows.

Understanding of secure operations and handling of privileged accounts.

Good-to-Have

Experience with CIS Benchmarks for RHEL and Windows.

Knowledge of log management/SIEM integration.

Understanding of operational technology networks and remote plant architectures.

Certifications (Preferable but Not Mandatory)

RHCSA / RHCE

VMware Certification for Carbon Black EDR