Data Engineer (AWS, Databricks)

Role and Responsibilities

Operational Responsibilities:

. Monitor and maintain production data pipelines to ensure 99.9% uptime and optimal performance

. Implement comprehensive logging, alerting, and monitoring systems using Application monitoring tools

. Perform regular health checks performance, job execution times, and resource utilization to identify and resolve bottlenecks proactively

. Manage incident response procedures for pipeline failures, including root cause analysis, resolution, and post-incident reviews

. Establish and maintain disaster recovery procedures and backup strategies for critical data assets within the Databricks environment

. Conduct regular performance tuning of Spark jobs and Databricks cluster configurations to optimize cost and execution efficiency
. Maintain comprehensive documentation for operational procedures, runbooks, and troubleshooting guides

. Coordinate scheduled maintenance windows and system upgrades with minimal business impact

. Manage user access controls, workspace configurations, and security policies within Application environments

Requirements / Qualifications

Education & Experience:

. Degree in Computer Science or Computer Engineering

. Minimum 5 years working experience in system operations compliance and management areas

. Project hands-on experience specifically with AWS platform (primary requirement)

. Project experience in cloud operations or cloud architecture

. Must be cloud certified (AWS)

Core Technical Skills:

. Proficiency in Databricks platform, including workspace management, cluster configuration, and job orchestration

. Strong expertise in Apache Spark within Databricks environment, including Spark SQL, DataFrames, and RDDs

. Good in-depth understanding of data warehouse concepts, data profiling, data verification and advanced analytics techniques

. Strong knowledge of monitoring, incident management, and cloud cost control

Technology Stack Experience:

. Databricks

. AWS cloud services and architecture

. IDMC (Informatica Data Management Cloud)

. Tableau for data visualization

. Oracle Database management

. ML Ops practices within Databricks environment

. STATA for statistical analysis is advantage

. Amazon SageMaker integration with Databricks

. DataRobot platform integration

Soft Skills & Stakeholder Management:

. Good interpersonal skills with the ability to work with different groups of stakeholders

. Strong problem-solving skills and ability to work independently in a fast-paced environment with minimal supervision

. Excellent communication skills for technical documentation and cross team collaboration

Desirable Requirements:

. AWS certification (Associate or Professional level) - highly preferred . Exposure to hospital information/clinical systems is an added advantage

. Understanding of DevOps practices and CI/CD pipelines for Databricks based data engineering projects