Data Engineer

Deegit Asia Pte Ltd is looking for a highly motivated and detail-oriented Data Engineer with 6 to 7 years of experience to join our growing analytics team. The ideal candidate will have strong analytical skills, hands-on experience in Databricks, Pyspark, AWS and a passion for data-driven decision making. You will play a key role in designing data pipelines and data storage solutions, including databases, data lakes, and warehouses.

Overview

Use comprehensive modern data engineering techniques and methods with Advanced Analytics to support business decisions for clients. can collect, aggregate, and analyze structured/unstructured data from multiple internal and external sources and patterns, insights, and trends to decision-makers. Your goal is to support the use of data-driven insights to help our Clients achieve business outcomes and objectives. In this role, you will be responsible for extracting, analyzing, and interpreting large datasets to support strategic business decisions. Proficiency in Pyspark, Databricks and data analytics is essential, as you will work closely with stakeholders to design data transformations , and orchestrating pipelines

. Work across workstreams to support data requirements including reports and dashboards

. Analyze and perform data profiling to understand data patterns and discrepancies following Data Quality and Data Management processes

. Understand and follow best practices to design and develop the E2E Data Pipeline: data transformation, ingestion, processing, and surfacing of data for large-scale applications

. Develop data pipeline automation using Azure, AWS data platform and technologies stack, Databricks, Data Factory

. Understand business requirements to translate them into technical requirements that the system analysts and other technical team members can drive into the project design and delivery

. Analyze source data and perform data ingestion in both batch and real-time patterns via various methods for example, file transfer, API, Data Streaming using Kafka and Spark Streaming

. Analyze and understand data processing and standardization requirements, develop ETL using Spark processing to transform data

. Understand data/reports and dashboards requirements, develop data export, data API, or data visualization using Power BI, Tableau, or other visualization tools

Key Accountabilities

. Work with large datasets

. Use Pyspark for data wrangling, transformation, and statistical analysis.

. Build reports, dashboards, and visualizations to present findings to stakeholders.

. Collaborate with cross-functional teams including business, product, and engineering to support analytics needs.

. Assist in developing data models and data pipelines as needed.

. Ensure data accuracy and consistency through validation and quality checks.

. Investigate and fix performance issues

. Travel as needed.

Key Role Capabilities Requirement

. Bachelor's degree in Computer Science, Computer Engineer, IT, or related fields

. Minimum of 4 years experience in Data Engineering fields

. Data Engineering skills: Python, SQL, Spark, Cloud Architect, Data & Solution Architect, API, Databricks, Azure, AWS

. Data Visualization skills: Power BI (or other visualization tools), DAX programming, API, Data Model, SQL, Story Telling and wireframe design

. Business Analyst skills: business knowledge, data profiling, basic data model design, data analysis, requirement analysis, SQL programing

. Basic knowledge in Data Lake/Data Warehousing/ Big data tools, Apache Spark, RDBMS and NoSQL, Knowledge Graph

. Experience working in a Singapore public sector, client-facing/consulting environment is a plus

. Team player, analytical and problem-solving skills

Education Qualifications

Bachelor's degree in computer science or other relevant fields.

Equivalent combination of Education and experience will also suffice.