Data Engineer

1. Data Pipeline Development & Management

. Design, build, and maintain robust data pipelines usingAWS Glue

. Implement ETL/ELT processes for data ingestion frommultiple sources

. Optimize data workflows for performance and scalability

. Monitor and troubleshoot data pipeline failures andperformance issues

2. Data Infrastructure & Engineering (with IPOS ITDepartment)

. Manage and optimize AWS Redshift data warehouse operations

. Configure and maintain data storage solutions (AWS S3,data lakes)

. Implement data partitioning, indexing, and compressionstrategies

. Support Infrastructure as Code (IaC) for datainfrastructure deployment

3. CI/CD & DevOps for Data (with AWS partners and IPOS ITDepartment)

. Develop and maintain CI/CD pipelines for data workflowsusing GitLab

. Implement automated testing for data pipelines and dataquality

. Support version control and deployment strategies for dataassets

. Configure AWS Lambda functions for data processingautomation

4. Monitoring & Support (with IPOS IT Department)

. Set up monitoring and alerting for data pipeline health

. Provide technical support for data-related issues

. Collaborate with technical teams on data architecturerequirements

. Optimize query performance and database operations

5. Documentation & Reporting (with IPOS IT Department)

. Document data pipeline architectures and technicalspecifications

. Maintain runbooks and operational procedures

. Conduct monthly progress meetings (1 hour) to report onsystem health

. Track engineering tasks through SHIP-HATS Jira

. Maintain technical documentation on SHIP-HATS Confluence

RequiredSkills & Experience:

. Strong background in data engineering and data pipelinedevelopment

. Proficiency in SQL, Python, and shell scripting

. Extensive experience with AWS data services (Redshift, S3,Glue, Lambda, CloudWatch)

. Data warehouse design and optimization experience

. Strong CI/CD pipeline knowledge (GitLab preferred)

. Infrastructure as Code (IaC) experience (Terraform,CloudFormation)

. Knowledge of data modeling and database design principles

. Strong troubleshooting and performance optimization skills

Thankyou!