Key Responsibilities
- Design, develop, and deploy data tables, views, and marts in data warehouses, operational data stores, data lakes, and data virtualization.
- Perform data extraction, cleaning, transformation, and data flow management. Web scraping may also be a part of the work scope in data extraction.
- Design, build, launch, and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks.
- Integrate and collate data silos in a manner that is both scalable and compliant.
- Collaborate with Project Managers, Data Architects, Business Analysts, Frontend Developers, Designers, and Data Analysts to build scalable, data-driven products.
- Be responsible for developing backend APIs and working on databases to support applications.
- Work in an Agile environment that practices Continuous Integration and Continuous Delivery.
- Work closely with fellow developers through pair programming and code review processes.
Experience and Skills Needed
- Proficient in general data cleaning and transformation (e.g., SQL, pandas, R, etc.) to ensure data accuracy and consistency.
- Proficient in building ETL pipelines (e.g., SQL Server Integration Services (SSIS), AWS Database Migration Service (DMS), Python, AWS Lambda, ECS Container Tasks, EventBridge, AWS Glue, Spring).
- Proficient in database design and various databases (e.g., SQL, PostgreSQL, AWS S3, Athena, MongoDB, PostGIS, MySQL, SQLite, VoltDB, Cassandra, etc.).
- Experience in cloud technologies such as GCP, GCC (i.e., AWS, Azure, Google Cloud).
- Experience and passion for data engineering in a big data environment using cloud platforms such as GCP, GCC (i.e., AWS, Azure, Google Cloud).
- Experience with building production-grade data pipelines and ETL/ELT data integration.
- Knowledge of system design, data structures, and algorithms.
- Familiar with data modelling, data access, and data storage infrastructure such as Data Marts, Data Lakes, Data Virtualization, and Data Warehouses for efficient storage and retrieval.
- Familiar with REST APIs and web requests/protocols in general.
- Familiar with big data frameworks and tools (e.g., Hadoop, Spark, Kafka, RabbitMQ).
- Familiar with W3C Document Object Model and customised web scraping (e.g., BeautifulSoup, CasperJS, PhantomJS, Selenium, Node.js, etc.).
- Familiar with data governance policies, access control, and security best practices.
- Comfortable with at least one scripting language (e.g., SQL, Python).
- Comfortable working in both Windows and Linux development environments.
- Interest in being the bridge between engineering and analytics.
Bonus Experience (Added Advantage)
- Experience building data engineering pipelines that require integration with search indexes.
- Experience with Airflow and RDBMS integration and implementation (e.g., MySQL).
- Experience with either Snowflake, Databricks, or an equivalent provider
