Data Engineer

Responsibility:

• Design, develop and deploy data tables , views and marts in data warehouses, operational data store, data lake and data virtualization.
• Perform data extraction, cleaning, transformation and flow . Web scraping may be also a part of the work scope in data extraction.
• Design, build, launch and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks.
• Integrate and collate data silos in a manner which is both scalable and compliant .
• Collaborate with Project Manager , Data Architect, Business Analysts, Frontend developers, Designers and Data analysts to build scalable data-driven products.
• Be responsible for developing backend APIs & working on databases to support the applications.
• Work in an Agile Environment that practices Continuous Integration and Delivery.
• Work closely with fellow developers through pair programming and code review process.
• Proficient in general data cleaning and transformation (e.g. SQL, pandas, R etc.,) to ensure data accuracy and consistency.
• Proficient in building ETL pipeline (eg. SQL Server Integration services(SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS Container task, Eventbridge, AWS Glue , Spring)
• Proficient in database design and various databases (e.g SQL, Postgre SQL,AWS S3, Athena , mongodb, Postgres/gis, mysql, sqlite, voltdb, cassandra, etc.,)
• Experience in cloud technologiessuch as GPC, GCC, (i.e.,AWS, Azure, Google Cloud)
• Experience and passion for data engineering in big data environment using cloud platforms such as GPC, GCC (i.e., AWS , Azure, Google cloud)
• Experience with building production-grade data pipelines, ETL/ELT data migration.
• Knowledge about system design , data structure and algorithms.
• Familiar with data modelling, data access, and data storage infrastructure like Data Mart, Data Lake, Data Virtualization and Data Warehouse for efficient storage and retrieval.
• Familiar with rest api and web requests/protocols in general.
• Familiar with big data frameworks and tools (eg. Hadoop, Spark, Kafka, RabbitMQ)
• Familiar with W3C Document Object Model and customised web scraping (e.g BeautifulSoup, CasperJS, PhantomJS, Selenium, NodeJs etc.,)
• Familiar with data governance policies, access control and security best practices.
• Comfortable in at least one scripting language (eg. SQL, Python)
• Comfortable with both Windows and Linux development environments.
• Interest in being the bridge between engineering and analytics.

Thanks, and Best Regards

Lini

Recruitment Consultant

R22108463