Senior Data Engineer

• 15-month contract
• Government project
• Hybrid work arrangement

Responsibilities:

• Enhance the existing data pipeline using industry-standard tools and mechanisms.
• Redesign and implement data pipeline to use AWS managed components.
• Implement data pipeline monitoring to alert us if there is any issue with the pipeline.
• Investigate and resolve any issue with data pipeline.
• Document the data pipeline solution and how to operate the data pipeline.
• Implement unit testing and integration testing for data pipeline component.

Requirements:

• At least 3 years hands-on working experience in data engineering, designing and building ETL pipelines.
• Experience in project execution and demonstrated technical expertise.
• Proficient in Python.
• SQL Knowledge.
• Schell scripts (Linux).
• Airflow or similar tool to create pipelines.
• AWS tools like S3, Glue, AWS EMR, Lambda, Step functions.
• Familiar with tools like AWS Batch, Athena, file formats like Parquet, NoSQL knowledge and tools like AWS OpenSearch.