Senior Data Engineer

1. Data Pipeline Development & Management

• Design, build, and maintain robust data pipelines using AWS Glue

• Implement ETL/ELT processes for data ingestion from multiple sources

• Optimize data workflows for performance and scalability

• Monitor and troubleshoot data pipeline failures and performance issues

2. Data Infrastructure & Engineering (with IT Department)

• Manage and optimize AWS Redshift data warehouse operations

• Configure and maintain data storage solutions (AWS S3, data lakes)

• Implement data partitioning, indexing, and compression strategies

• Support Infrastructure as Code (IaC) for data infrastructure deployment

3. CI/CD & DevOps for Data (with AWS partners and IT Department)

• Develop and maintain CI/CD pipelines for data workflows using GitLab

• Implement automated testing for data pipelines and data quality

• Support version control and deployment strategies for data assets

• Configure AWS Lambda functions for data processing automation

4. Monitoring & Support (with IT Department)

• Set up monitoring and alerting for data pipeline health

• Provide technical support for data-related issues

• Collaborate with technical teams on data architecture requirements

• Optimize query performance and database operations

5. Documentation & Reporting (with IT Department)

• Document data pipeline architectures and technical specifications

• Maintain runbooks and operational procedures

• Conduct monthly progress meetings (1 hour) to report on system health

• Track engineering tasks through SHIP-HATS Jira

• Maintain technical documentation on SHIP-HATS Confluence

Required Skills & Experience:

• Strong background in data engineering and data pipeline development

• Proficiency in SQL, Python, and shell scripting

• Extensive experience with AWS data services (Redshift, S3, Glue, Lambda, CloudWatch)

• Data warehouse design and optimization experience

• Strong CI/CD pipeline knowledge (GitLab preferred)

• Infrastructure as Code (IaC) experience (Terraform, CloudFormation)

• Knowledge of data modeling and database design principles

• Strong troubleshooting and performance optimization skills