Your role
Our major Oil & Gas Downstream Client is urgently looking for Data Engineer who will be responsible for identifying relevant data to be gathered and implementing a solution to do so, including knowledge of where to get the data, how to extract it in the right format, and how to augment the data to support data science/analysis requirements. You need to be able to build a robust, fault-tolerant data pipeline that cleans, transforms, and aggregates un-organized and messy data into databases or data sources. You need to be able to ensure that data flows smoothly from source to destination so that it can be processed.
You will support our software developers, database architects, data analysts and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. You must be self-directed and comfortable supporting the data needs of multiple teams, systems, and products
You will be defining and building the data pipelines that will enable faster, better, data-informed decision-making within the business, assisting in developing enterprise data management strategies including operational data stores, data warehouse implementations, data movement, data services, data acquisition, data conversion, and archive/recovery. Moreover, design and develop Data Products to drive business outcomes.
Your accountabilities include the following:
- Work together with the solution and data Architects to design the next generation data platform to support the Analytics in Trading.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Assemble large, complex data sets that meet functional / non-functional business requirements, which are scalable and easy to maintain.
- Build and maintain data pipelines required for extracting, cleaning, and transforming data from a wide variety of sources.
- Help establish best practices around data governance, data security- and privacy (GDPR).
- Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
- Drives and applies Agile Framework within your operating environment
You are what we are looking for if you:
- Have at least a year of experience in a Data Engineer role with following preferred skills and experience:
- Have 3-5 years of advanced experience in design and coding skills and programming experience in Python, Pyspark, SQL
- Have experience with working concepts as data pipelining / data wrangling. Strong business analysis skills to obtain requirements from users and document them.
- Have experience in interpretation of data, highly analytical, troubleshooting, and problem-solving skills. Experience with data architecture and design techniques (local/abstract).
- Have the following technical skills: Source Code Control (Git, Subversion), reporting/visualization tools (Power BI), SQL-based technologies (MSSQL, MySQL, PostgreSQL), restful API developments, data blending technology (Databricks preferred), Microsoft data stack (Microsoft Azure Data Lake, Azure Table/Blob stores etc.)
- Have experience in Agile Project Delivery techniques (e.g. SCRUM) & experience creating and using a wide range of scrum and agile artefacts and techniques, automated testing, personas, user stories, agile games etc. Awareness of Agile Framework and its tools such as Visual Studio Team Services (VSTS).
- Have interpersonal, communication, facilitation, and presentation skills. Is experienced in communicating with business at all levels
- Are comfortable to work with global stakeholders, internally and externally. Experience of working in a global business environment/different cultures.
- Are able to work under pressure with tight deadlines and balancing multiple priorities.
