In your new role you will:
- Design, build and maintain scalable and secure infrastructure for data engineering workloads.
- Develop and manage CI/CD pipelines for data pipelines and software deployment.
- Collaborate with data engineers and data scientists to streamline data ingestion, processing, and storage.
- Automate deployment, monitoring and alerting for data services.
- Ensure system security, availability and performance across datapipelines and services.
- Manage containerised environments using Docker and Kubernetes. Knowledge of Red Hat OpenShift is a plus.
- Optimise data workflows and troubleshoot infrastructure or pipeline issues.
- Monitor and maintain cloud resources and optimise costs.
You are best equipped for this task if you have:
- Minimum Bachelor's degree in Computer Science, Engineering or related field.
- 1-2 years of experience in DevOps, Data Engineering or similar roles.
- Familiarity with DevOps, CI/CD, containerization and data orchestration concepts.
- Experience with at least one major cloud services provider (Amazon/Azure/GCP).
- Knowledge of modern data orchestration tools like Apache Airflow/Dagster/Prefect etc.
- Knowledge of modern data transformation frameworks like DBT (DataBuild Tool) etc.
- Solid understanding of Python and SQL.
- Solid understanding of data warehouse and data lake architectures(e.g. Redshift, Snowflake, Databricks etc).
