About the Role
As a Senior Data Engineer, you will focus on designing, building, and maintaining scalable data solutions that support business needs and AI applications. You will play a key role in:
Data Pipeline & Architecture Development
- Build and optimize automated data pipelines for ingesting, transforming, and processing large datasets.
- Design efficient data architectures that support analytics, machine learning, and real-time applications.
Cloud Migration & AI Enablement
- Support cloud migration efforts, transitioning on-premises data workflows to cloud-based platforms like Databricks.
- Collaborate with data scientists to improve feature selection, feature engineering, and enable end-to-end AI workflows from model training to deployment and monitoring.
CI/CD & Automation
- Develop CI/CD pipelines to streamline data pipeline deployments and ensure stable, automated workflows.
- Improve monitoring and observability to maintain system reliability.
Collaboration & Business Impact
- Work with data scientists, product teams, and platform engineers to align data solutions with business objectives.
- Ensure data quality, security, and compliance with industry standards.
- Contribute to best practices in data governance, documentation, and automation.
Qualifications & Skills
- Degree holder of Information Technology, Mathematics or Statistics with least 3 years of experience in data engineering.
- Expert in Python, Java, SQL, Linux Shell.
- Experience in UNIX environment, Git Flow, CI/CD automation, Jenkins, Bitbucket.
- Hands-on experience with Spark, Hadoop platforms & tools (Hive, Impala, Airflow, NiFi), Spring Boot, etc. to build big data products & platforms.
- Proficiency with a modern cloud or hybrid-cloud stack (AWS, Databricks, Cloudera, etc).
- Experience in building and deploying production-level data-driven applications and data processing workflows or pipelines.
