Data Engineer

Job Description:

-Design, develop and deploy data tables, views and marts in data warehouses,operational data store, data lake and data virtualization.

-Perform data extraction, cleaning, transformation, and flow. Web scraping maybe also a part of the work scope in data extraction.

-Design, build, launch and maintain efficient and reliable large-scale batch andreal-time data pipelines with data processing frameworks.

-Integrate and collate data silos in a manner which is both scalable andcompliant.

-Collaborate with Project Manager, Data Architect, Business Analysts, FrontendDevelopers, Designers and Data Analyst to build scalable data driven products.

- Beresponsible for developing backend APIs & working on databases to supportthe applications.

- Workin an Agile Environment that practices Continuous Integration and Delivery.

- Workclosely with fellow developers through pair programming and code reviewprocess.

The teamis expected to perform Data Warehousing tasks, mainly in AWS GCC, and manageAPIs

Qualifications

-Proficient in general data cleaning and transformation (e.g. SQL, pandas, R,etc) to ensure data accuracy and consistency.

-Proficient in building ETL pipeline (e.g. SQL Server Integration Services

-(SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECSContainer task, Eventbridge, AWS Glue, Spring).

-Proficient in database design and various databases (e.g. SQL, PostgreSQL, AWSS3, Athena, MongoDB, postgres/gis, MySQL, SQLite, voltdb, Cassandra, etc).

-Experience in cloud technologies such as GPC, GCC (i.e. AWS, Azure, GoogleCloud).

-Experience and passion for data engineering in a big data environment usingCloud platforms such as GPC, GCC (i.e. AWS, Azure, Google Cloud).

-Experience with building production-grade data pipelines, ETL/ELT dataintegration.

-Knowledge about system design, data structure and algorithms.

-Familiar with data modelling, data access, and data storage infrastructure likeData Mart, Data Lake, Data Virtualisation and Data Warehouse for efficientstorage and retrieval.

-Familiar with rest api and web requests/protocols in general.

-Familiar with big data frameworks and tools (eg. Hadoop, Spark, Kafka, RabbitMQ).

-Familiar with W3C Document Object Model and customized web scraping (e.g.BeautifulSoup, CasperJS, PhantomJS, Selenium, Nodejs, etc).

-Familiar with data governance policies, access control and security bestpractices.

-Comfortable in at least one scripting language (eg. SQL, Python).

-Comfortable in both windows and Linux development environments.

-Interest in being the bridge between engineering and analytics