Job Responsibilities:
- Work across workstreams to support data requirements including reports and dashboards
- Analyze and perform data profiling to understand data patterns and discrepancies following Data Quality and Data Management processes
- Understand and follow best practices to design and develop the E2E Data Pipeline: data transformation, ingestion, processing, and surfacing of data for large-scale applications
- Develop data pipeline automation using Azure, AWS data platform and technologies stack, Databricks, Data Factory
- Understand business requirements to translate them into technical requirements that the system analysts and other technical team members can drive into the project design and delivery
- Analyze source data and perform data ingestion in both batch and real-time patterns via various methods for example, file transfer, API, Data Streaming using Kafka and Spark Streaming
- Analyze and understand data processing and standardization requirements, develop ETL using Spark processing to transform data
- Understand data/reports and dashboards requirements, develop data export, data API, or data visualization using Power BI, Tableau, or other visualization tools
Qualification:
We are looking for experience and qualifications in the following:
- Bachelor's degree in Computer Science, Computer Engineer, IT, or related fields
- Minimum of 4 years experience in Data Engineering fields
- Data Engineering skills: Python, SQL, Spark, Cloud Architect, Data & Solution Architect, API, Databricks, Azure, AWS
- Data Visualization skills: Power BI (or other visualization tools), DAX programming, API, Data Model, SQL, Story Telling and wireframe design
- Business Analyst skills: business knowledge, data profiling, basic data model design, data analysis, requirement analysis, SQL programing
- Basic knowledge in Data Lake/Data Warehousing/ Big data tools, Apache Spark, RDBMS and NoSQL, Knowledge Graph
- Experience working in a Singapore public sector, client-facing/consulting environment is a plus
- Team player, analytical and problem-solving skills
Mandatory Skills: PySpark, Databricks
