Data Engineer

EXPERIENCE AND SKILLS NEEDED

- Proficient in general data cleaning and transformation (e.g. SQL, pandas, R, etc) to ensure data accuracy and consistency.

- Proficient in building ETL pipeline (eg. SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS Container task, Eventbridge, AWS Glue, Spring).

- Proficient in database design and various databases (e.g. SQL, PostgreSQL, AWS S3, Athena, mongodb, postgres/gis, mysql, sqlite, voltdb, cassandra, etc).

- Experience in cloud technologies such as GPC, GCC (i.e. AWS, Azure, Google Cloud).

- Experience and passion for data engineering in a big data environment using Cloud platforms such as GPC, GCC (i.e. AWS, Azure, Google Cloud).

- Experience with building production-grade data pipelines, ETL/ELT data integration.

- Knowledge about system design, data structure and algorithms.

- Familiar with data modelling, data access, and data storage infrastructure like Data Mart, Data Lake, Data Virtualisation and Data Warehouse for efficient storage and retrieval.

- Familiar with rest api and web requests/protocols in general.

- Familiar with big data frameworks and tools (eg. Hadoop, Spark, Kafka,RabbitMQ).

- Familiar with W3C Document Object Model and customized web scraping (e.g. BeautifulSoup, CasperJS, PhantomJS, Selenium, Nodejs, etc).

- Familiar with data governance policies, access control and security best practices.

- Comfortable in at least one scripting language (eg. SQL,Python).

- Comfortable in both windows and linux development environments.

- Interest in being the bridge between engineering and analytics.

Bonus Experience (Added Advantage):

- Have experience building data engineering pipelines that requires integration with search indexes and is better

- Have experience with Airflow and RDBMS integration and implementation (e.g. MySQL)